“You are running for food, or you are running from becoming food. And often times, you can’t tell which. Either way, run.” NVIDIA founder and CEO Jensen Huang today urged graduates of National Taiwan University to run hard to seize the unprecedented opportunities that AI will present, but embrace the inevitable failures along the way. Read article >
( 5
min )
The primary goal in recommendation is to suggest relevant content to users,
but optimizing for accuracy often results in recommendations that lack
diversity. To remedy this, conventional approaches such as re-ranking improve
diversity by presenting more diverse items. Here we argue that to promote
inherent and prolonged diversity, the system must encourage its creation.
Towards this, we harness the performative nature of recommendation, and show
how learning can incentivize strategic content creators to create diverse
content. Our approach relies on a novel form of regularization that anticipates
strategic changes to content, and penalizes for content homogeneity. We provide
analytic and empirical results that demonstrate when and how diversity can be
incentivized, and experimentally demonstrate the utility of our approach on
synthetic and semi-synthetic data.
( 2
min )
This paper delves into stochastic optimization problems that involve
Markovian noise. We present a unified approach for the theoretical analysis of
first-order gradient methods for stochastic optimization and variational
inequalities. Our approach covers scenarios for both non-convex and strongly
convex minimization problems. To achieve an optimal (linear) dependence on the
mixing time of the underlying noise sequence, we use the randomized batching
scheme, which is based on the multilevel Monte Carlo method. Moreover, our
technique allows us to eliminate the limiting assumptions of previous research
on Markov noise, such as the need for a bounded domain and uniformly bounded
stochastic gradients. Our extension to variational inequalities under Markovian
noise is original. Additionally, we provide lower bounds that match the oracle
complexity of our method in the case of strongly convex optimization problems.
( 2
min )
The increasing adoption of text-to-speech technologies has led to a growing
demand for natural and emotive voices that adapt to a conversation's context
and emotional tone. The Emotive Narrative Storytelling (EMNS) corpus is a
unique speech dataset created to enhance conversations' expressiveness and
emotive quality in interactive narrative-driven systems. The corpus consists of
a 2.3-hour recording featuring a female speaker delivering labelled utterances.
It encompasses eight acted emotional states, evenly distributed with a variance
of 0.68%, along with expressiveness levels and natural language descriptions
with word emphasis labels. The evaluation of audio samples from different
datasets revealed that the EMNS corpus achieved the highest average scores in
accurately conveying emotions and demonstrating expressiveness. It outperformed
other datasets in conveying shared emotions and achieved comparable levels of
genuineness. A classification task confirmed the accurate representation of
intended emotions in the corpus, with participants recognising the recordings
as genuine and expressive. Additionally, the availability of the dataset
collection tool under the Apache 2.0 License simplifies remote speech data
collection for researchers.
( 2
min )
Federated learning algorithms are developed both for efficiency reasons and
to ensure the privacy and confidentiality of personal and business data,
respectively. Despite no data being shared explicitly, recent studies showed
that the mechanism could still leak sensitive information. Hence, secure
aggregation is utilized in many real-world scenarios to prevent attribution to
specific participants. In this paper, we focus on the quality of individual
training datasets and show that such quality information could be inferred and
attributed to specific participants even when secure aggregation is applied.
Specifically, through a series of image recognition experiments, we infer the
relative quality ordering of participants. Moreover, we apply the inferred
quality information to detect misbehaviours, to stabilize training performance,
and to measure the individual contributions of participants.
( 2
min )
Energy markets can provide incentives for undesired behavior of market
participants. Multi-agent Reinforcement learning (MARL) is a promising new
approach to predicting the expected behavior of energy market participants.
However, reinforcement learning requires many interactions with the system to
converge, and the power system environment often consists of extensive
computations, e.g., optimal power flow (OPF) calculation for market clearing.
To tackle this complexity, we provide a model of the energy market to a basic
MARL algorithm in the form of a learned OPF approximation and explicit market
rules. The learned OPF surrogate model makes an explicit solving of the OPF
completely unnecessary. Our experiments demonstrate that the model additionally
reduces training time by about one order of magnitude but at the cost of a
slightly worse approximation of the Nash equilibrium. Potential applications of
our method are market design, more realistic modeling of market participants,
and analysis of manipulative behavior.
( 2
min )
With the freight delivery demands and shipping costs increasing rapidly,
intelligent control of fleets to enable efficient and cost-conscious solutions
becomes an important problem. In this paper, we propose DeepFreight, a
model-free deep-reinforcement-learning-based algorithm for multi-transfer
freight delivery, which includes two closely-collaborative components:
truck-dispatch and package-matching. Specifically, a deep multi-agent
reinforcement learning framework called QMIX is leveraged to learn a dispatch
policy, with which we can obtain the multi-step joint vehicle dispatch
decisions for the fleet with respect to the delivery requests. Then an
efficient multi-transfer matching algorithm is executed to assign the delivery
requests to the trucks. Also, DeepFreight is integrated with a Mixed-Integer
Linear Programming optimizer for further optimization. The evaluation results
show that the proposed system is highly scalable and ensures a 100\% delivery
success while maintaining low delivery-time and fuel consumption. The codes are
available at https://github.com/LucasCJYSDL/DeepFreight.
( 2
min )
Unsupervised disentanglement is a long-standing challenge in representation
learning. Recently, self-supervised techniques achieved impressive results in
the sequential setting, where data is time-dependent. However, the latter
methods employ modality-based data augmentations and random sampling or solve
auxiliary tasks. In this work, we propose to avoid that by generating,
sampling, and comparing empirical distributions from the underlying variational
model. Unlike existing work, we introduce a self-supervised sequential
disentanglement framework based on contrastive estimation with no external
signals, while using common batch sizes and samples from the latent space
itself. In practice, we propose a unified, efficient, and easy-to-code sampling
strategy for semantically similar and dissimilar views of the data. We evaluate
our approach on video, audio, and time series benchmarks. Our method presents
state-of-the-art results in comparison to existing techniques. The code is
available at https://github.com/azencot-group/SPYL.
( 2
min )
Some of the most successful knowledge graph embedding (KGE) models for link
prediction -- CP, RESCAL, TuckER, ComplEx -- can be interpreted as energy-based
models. Under this perspective they are not amenable for exact
maximum-likelihood estimation (MLE), sampling and struggle to integrate logical
constraints. This work re-interprets the score functions of these KGEs as
circuits -- constrained computational graphs allowing efficient
marginalisation. Then, we design two recipes to obtain efficient generative
circuit models by either restricting their activations to be non-negative or
squaring their outputs. Our interpretation comes with little or no loss of
performance for link prediction, while the circuits framework unlocks exact
learning by MLE, efficient sampling of new triples, and guarantee that logical
constraints are satisfied by design. Furthermore, our models scale more
gracefully than the original KGEs on graphs with millions of entities.
( 2
min )
We consider a version of actor-critic which uses proportional step-sizes and
only one critic update with a single sample from the stationary distribution
per actor step. We provide an analysis of this method using the small-gain
theorem. Specifically, we prove that this method can be used to find a
stationary point, and that the resulting sample complexity improves the state
of the art for actor-critic methods to $O \left(\mu^{-2} \epsilon^{-2} \right)$
to find an $\epsilon$-approximate stationary point where $\mu$ is the condition
number associated with the critic.
( 2
min )
In recent years, numerous screening methods have been published for
ultrahigh-dimensional data that contain hundreds of thousands of features;
however, most of these features cannot handle data with thousands of classes.
Prediction models built to authenticate users based on multichannel biometric
data result in this type of problem. In this study, we present a novel method
known as random forest-based multiround screening (RFMS) that can be
effectively applied under such circumstances. The proposed algorithm divides
the feature space into small subsets and executes a series of partial model
builds. These partial models are used to implement tournament-based sorting and
the selection of features based on their importance. To benchmark RFMS, a
synthetic biometric feature space generator known as BiometricBlender is
employed. Based on the results, the RFMS is on par with industry-standard
feature screening methods while simultaneously possessing many advantages over
these methods.
( 2
min )
A crucial problem in reinforcement learning is learning the optimal policy.
We study this in tabular infinite-horizon discounted Markov decision processes
under the online setting. The existing algorithms either fail to achieve regret
optimality or have to incur a high memory and computational cost. In addition,
existing optimal algorithms all require a long burn-in time in order to achieve
optimal sample efficiency, i.e., their optimality is not guaranteed unless
sample size surpasses a high threshold. We address both open problems by
introducing a model-free algorithm that employs variance reduction and a novel
technique that switches the execution policy in a slow-yet-adaptive manner.
This is the first regret-optimal model-free algorithm in the discounted
setting, with the additional benefit of a low burn-in time.
( 2
min )
A key challenge for a reinforcement learning (RL) agent is to incorporate
external/expert1 advice in its learning. The desired goals of an algorithm that
can shape the learning of an RL agent with external advice include (a)
maintaining policy invariance; (b) accelerating the learning of the agent; and
(c) learning from arbitrary advice [3]. To address this challenge this paper
formulates the problem of incorporating external advice in RL as a multi-armed
bandit called shaping-bandits. The reward of each arm of shaping bandits
corresponds to the return obtained by following the expert or by following a
default RL algorithm learning on the true environment reward.We show that
directly applying existing bandit and shaping algorithms that do not reason
about the non-stationary nature of the underlying returns can lead to poor
results. Thus we propose UCB-PIES (UPIES), Racing-PIES (RPIES), and Lazy PIES
(LPIES) three different shaping algorithms built on different assumptions that
reason about the long-term consequences of following the expert policy or the
default RL algorithm. Our experiments in four different settings show that
these proposed algorithms achieve the above-mentioned goals whereas the other
algorithms fail to do so.
( 2
min )
Convolutional Neural Network (CNN) is one of the most important architectures
in deep learning. The fundamental building block of a CNN is a trainable
filter, represented as a discrete grid, used to perform convolution on discrete
input data. In this work, we propose a continuous version of a trainable
convolutional filter able to work also with unstructured data. This new
framework allows exploring CNNs beyond discrete domains, enlarging the usage of
this important learning technique for many more complex problems. Our
experiments show that the continuous filter can achieve a level of accuracy
comparable to the state-of-the-art discrete filter, and that it can be used in
current deep learning architectures as a building block to solve problems with
unstructured domains as well.
( 2
min )
Recently, Montasser et al. [2019] showed that finite VC dimension is not
sufficient for proper adversarially robust PAC learning. In light of this
hardness, there is a growing effort to study what type of relaxations to the
adversarially robust PAC learning setup can enable proper learnability. In this
work, we initiate the study of proper learning under relaxations of the
worst-case robust loss. We give a family of robust loss relaxations under which
VC classes are properly PAC learnable with sample complexity close to what one
would require in the standard PAC learning setup. On the other hand, we show
that for an existing and natural relaxation of the worst-case robust loss,
finite VC dimension is not sufficient for proper learning. Lastly, we give new
generalization guarantees for the adversarially robust empirical risk
minimizer.
( 2
min )
We investigate the convergence of stochastic mirror descent (SMD) under
interpolation in relatively smooth and smooth convex optimization. In
relatively smooth convex optimization we provide new convergence guarantees for
SMD with a constant stepsize. For smooth convex optimization we propose a new
adaptive stepsize scheme -- the mirror stochastic Polyak stepsize (mSPS).
Notably, our convergence results in both settings do not make bounded gradient
assumptions or bounded variance assumptions, and we show convergence to a
neighborhood that vanishes under interpolation. Consequently, these results
correspond to the first convergence guarantees under interpolation for the
exponentiated gradient algorithm for fixed or adaptive stepsizes. mSPS
generalizes the recently proposed stochastic Polyak stepsize (SPS) (Loizou et
al. 2021) to mirror descent and remains both practical and efficient for modern
machine learning applications while inheriting the benefits of mirror descent.
We complement our results with experiments across various supervised learning
tasks and different instances of SMD, demonstrating the effectiveness of mSPS.
( 2
min )
Continuous monitoring with an ever-increasing number of sensors has become
ubiquitous across many application domains. However, acquired time series are
typically high-dimensional and difficult to interpret. Expressive deep learning
(DL) models have gained popularity for dimensionality reduction, but the
resulting latent space often remains difficult to interpret. In this work we
propose SOM-CPC, a model that visualizes data in an organized 2D manifold,
while preserving higher-dimensional information. We address a largely
unexplored and challenging set of scenarios comprising high-rate time series,
and show on both synthetic and real-life data (physiological data and audio
recordings) that SOM-CPC outperforms strong baselines like DL-based feature
extraction, followed by conventional dimensionality reduction techniques, and
models that jointly optimize a DL model and a Self-Organizing Map (SOM).
SOM-CPC has great potential to acquire a better understanding of latent
patterns in high-rate data streams.
( 2
min )
Identifiability of latent variable models has recently gained interest in
terms of its applications to interpretability or out of distribution
generalisation. In this work, we study identifiability of Markov Switching
Models as a first step towards extending recent results to sequential latent
variable models. We present identifiability conditions within first-order
Markov dependency structures, and parametrise the transition distribution via
non-linear Gaussians. Our experiments showcase the applicability of our
approach for regime-dependent causal discovery and high-dimensional time series
segmentation.
( 2
min )
Masked Language Models (MLMs) have proven to be effective for second-pass
rescoring in Automatic Speech Recognition (ASR) systems. In this work, we
propose Masked Audio Text Encoder (MATE), a multi-modal masked language model
rescorer which incorporates acoustic representations into the input space of
MLM. We adopt contrastive learning for effectively aligning the modalities by
learning shared representations. We show that using a multi-modal rescorer is
beneficial for domain generalization of the ASR system when target domain data
is unavailable. MATE reduces word error rate (WER) by 4%-16% on in-domain, and
3%-7% on out-of-domain datasets, over the text-only baseline. Additionally,
with very limited amount of training data (0.8 hours), MATE achieves a WER
reduction of 8%-23% over the first-pass baseline.
( 2
min )
Community detection is an important problem in unsupervised learning. This
paper proposes to solve a projection matrix approximation problem with an
additional entrywise bounded constraint. Algorithmically, we introduce a new
differentiable convex penalty and derive an alternating direction method of
multipliers (ADMM) algorithm. Theoretically, we establish the convergence
properties of the proposed algorithm. Numerical experiments demonstrate the
superiority of our algorithm over its competitors, such as the semi-definite
relaxation method and spectral clustering.
( 2
min )
Annotating data for multi-label classification is prohibitively expensive
because every category of interest must be confirmed to be present or absent.
Recent work on single positive multi-label (SPML) learning shows that it is
possible to train effective multi-label classifiers using only one positive
label per image. However, the standard benchmarks for SPML are derived from
traditional multi-label classification datasets by retaining one positive label
for each training example (chosen uniformly at random) and discarding all other
labels. In realistic settings it is not likely that positive labels are chosen
uniformly at random. This work introduces protocols for studying label bias in
SPML and provides new empirical results.
( 2
min )
We study online influence maximization (OIM) under a new model of decreasing
cascade (DC). This model is a generalization of the independent cascade (IC)
model by considering the common phenomenon of market saturation. In DC, the
chance of an influence attempt being successful reduces with previous failures.
The effect is neglected by previous OIM works under IC and linear threshold
models. We propose the DC-UCB algorithm to solve this problem, which achieves a
regret bound of the same order as the state-of-the-art works on the IC model.
Extensive experiments on both synthetic and real datasets show the
effectiveness of our algorithm.
( 2
min )
Machine learning has been applied to the problem of X-ray diffraction phase
prediction with promising results. In this paper, we describe a method for
using machine learning to predict crystal structure phases from X-ray
diffraction data of transition metals and their oxides. We evaluate the
performance of our method and compare the variety of its settings. Our results
demonstrate that the proposed machine learning framework achieves competitive
performance. This demonstrates the potential for machine learning to
significantly impact the field of X-ray diffraction and crystal structure
determination. Open-source implementation:
https://github.com/maxnygma/NeuralXRD.
( 2
min )
I study a stochastic multi-arm bandit problem where rewards are subject to
adversarial corruption. I propose a novel attack strategy that manipulates a
learner employing the UCB algorithm into pulling some non-optimal target arm $T
- o(T)$ times with a cumulative cost that scales as $\widehat{O}(\sqrt{\log
T})$, where $T$ is the number of rounds. I also prove the first lower bound on
the cumulative attack cost. The lower bound matches the upper bound up to
$O(\log \log T)$ factors, showing the proposed attack strategy to be near
optimal.
( 2
min )
We examine the characteristic activation values of individual ReLU units in
neural networks. We refer to the corresponding set for such characteristic
activation values in the input space as the characteristic activation set of a
ReLU unit. We draw an explicit connection between the characteristic activation
set and learned features in ReLU networks. This connection leads to new
insights into why various neural network normalization techniques used in
modern deep learning architectures regularize and stabilize SGD optimization.
Utilizing these insights, we propose a geometric approach to parameterize ReLU
networks for improved feature learning. We empirically verify its usefulness
with less carefully chosen initialization schemes and larger learning rates. We
report improved optimization stability, faster convergence speed, and better
generalization performance.
( 2
min )
Machine learning systems such as large scale recommendation systems or
natural language processing systems are usually trained on billions of training
points and are associated with hundreds of billions or trillions of parameters.
Improving the learning process in such a way that both the training load is
reduced and the model accuracy improved is highly desired. In this paper we
take a first step toward solving this problem, studying influence functions
from the perspective of simplifying the computations they involve. We discuss
assumptions, under which influence computations can be performed on
significantly fewer parameters. We also demonstrate that the sign of the
influence value can indicate whether a training point is to memorize, as
opposed to generalize upon. For this purpose we formally define what
memorization means for a training point, as opposed to generalization. We
conclude that influence functions can be made practical, even for large scale
machine learning systems, and that influence values can be taken into account
by algorithms that selectively remove training points, as part of the learning
process.
( 2
min )
This paper delves into stochastic optimization problems that involve
Markovian noise. We present a unified approach for the theoretical analysis of
first-order gradient methods for stochastic optimization and variational
inequalities. Our approach covers scenarios for both non-convex and strongly
convex minimization problems. To achieve an optimal (linear) dependence on the
mixing time of the underlying noise sequence, we use the randomized batching
scheme, which is based on the multilevel Monte Carlo method. Moreover, our
technique allows us to eliminate the limiting assumptions of previous research
on Markov noise, such as the need for a bounded domain and uniformly bounded
stochastic gradients. Our extension to variational inequalities under Markovian
noise is original. Additionally, we provide lower bounds that match the oracle
complexity of our method in the case of strongly convex optimization problems.
( 2
min )
The $L_{2}$-regularized loss of Deep Linear Networks (DLNs) with more than
one hidden layers has multiple local minima, corresponding to matrices with
different ranks. In tasks such as matrix completion, the goal is to converge to
the local minimum with the smallest rank that still fits the training data.
While rank-underestimating minima can easily be avoided since they do not fit
the data, gradient descent might get stuck at rank-overestimating minima. We
show that with SGD, there is always a probability to jump from a higher rank
minimum to a lower rank one, but the probability of jumping back is zero. More
precisely, we define a sequence of sets $B_{1}\subset B_{2}\subset\cdots\subset
B_{R}$ so that $B_{r}$ contains all minima of rank $r$ or less (and not more)
that are absorbing for small enough ridge parameters $\lambda$ and learning
rates $\eta$: SGD has prob. 0 of leaving $B_{r}$, and from any starting point
there is a non-zero prob. for SGD to go in $B_{r}$.
( 2
min )
Identifiability of latent variable models has recently gained interest in
terms of its applications to interpretability or out of distribution
generalisation. In this work, we study identifiability of Markov Switching
Models as a first step towards extending recent results to sequential latent
variable models. We present identifiability conditions within first-order
Markov dependency structures, and parametrise the transition distribution via
non-linear Gaussians. Our experiments showcase the applicability of our
approach for regime-dependent causal discovery and high-dimensional time series
segmentation.
( 2
min )
We consider the variable selection problem for two-sample tests, aiming to
select the most informative variables to distinguish samples from two groups.
To solve this problem, we propose a framework based on the kernel maximum mean
discrepancy (MMD). Our approach seeks a group of variables with a pre-specified
size that maximizes the variance-regularized MMD statistics. This formulation
also corresponds to the minimization of asymptotic type-II error while
controlling type-I error, as studied in the literature. We present
mixed-integer programming formulations and offer exact and approximation
algorithms with performance guarantees for linear and quadratic types of kernel
functions. Experimental results demonstrate the superior performance of our
framework.
( 2
min )
We study the problem of approximate sampling from non-log-concave
distributions, e.g., Gaussian mixtures, which is often challenging even in low
dimensions due to their multimodality. We focus on performing this task via
Markov chain Monte Carlo (MCMC) methods derived from discretizations of the
overdamped Langevin diffusions, which are commonly known as Langevin Monte
Carlo algorithms. Furthermore, we are also interested in two nonsmooth cases
for which a large class of proximal MCMC methods have been developed: (i) a
nonsmooth prior is considered with a Gaussian mixture likelihood; (ii) a
Laplacian mixture distribution. Such nonsmooth and non-log-concave sampling
tasks arise from a wide range of applications to Bayesian inference and imaging
inverse problems such as image deconvolution. We perform numerical simulations
to compare the performance of most commonly used Langevin Monte Carlo
algorithms.
( 2
min )
Recently, Montasser et al. [2019] showed that finite VC dimension is not
sufficient for proper adversarially robust PAC learning. In light of this
hardness, there is a growing effort to study what type of relaxations to the
adversarially robust PAC learning setup can enable proper learnability. In this
work, we initiate the study of proper learning under relaxations of the
worst-case robust loss. We give a family of robust loss relaxations under which
VC classes are properly PAC learnable with sample complexity close to what one
would require in the standard PAC learning setup. On the other hand, we show
that for an existing and natural relaxation of the worst-case robust loss,
finite VC dimension is not sufficient for proper learning. Lastly, we give new
generalization guarantees for the adversarially robust empirical risk
minimizer.
( 2
min )
Generative Adversarial Networks (GANs) have shown immense potential in fields
far from physics, such as in text and image generation. Here we use GANs to
learn a prototypical stochastic process on a lattice. By suitably adding noise
to the original data we succeed in bringing both the Generator and the
Discriminator loss functions close to their ideal value. However, as typical
for adversarial approaches, oscillations persist. This undermines model
selection and the quality of the generated trajectory. We demonstrate that a
suitable multi-model procedure where stochastic trajectories are advanced at
each step upon randomly selecting a Generator leads to a remarkable increase in
accuracy. Based on the reported findings GANs appears as a promising tool to
tackle complex statistical dynamics.
( 2
min )
Federated learning algorithms are developed both for efficiency reasons and
to ensure the privacy and confidentiality of personal and business data,
respectively. Despite no data being shared explicitly, recent studies showed
that the mechanism could still leak sensitive information. Hence, secure
aggregation is utilized in many real-world scenarios to prevent attribution to
specific participants. In this paper, we focus on the quality of individual
training datasets and show that such quality information could be inferred and
attributed to specific participants even when secure aggregation is applied.
Specifically, through a series of image recognition experiments, we infer the
relative quality ordering of participants. Moreover, we apply the inferred
quality information to detect misbehaviours, to stabilize training performance,
and to measure the individual contributions of participants.
( 2
min )
Multilabel ranking is a central task in machine learning. However, the most
fundamental question of learnability in a multilabel ranking setting with
relevance-score feedback remains unanswered. In this work, we characterize the
learnability of multilabel ranking problems in both batch and online settings
for a large family of ranking losses. Along the way, we give two equivalence
classes of ranking losses based on learnability that capture most, if not all,
losses used in practice.
( 2
min )
We apply the Hierarchical Autoregressive Neural (HAN) network sampling
algorithm to the two-dimensional $Q$-state Potts model and perform simulations
around the phase transition at $Q=12$. We quantify the performance of the
approach in the vicinity of the first-order phase transition and compare it
with that of the Wolff cluster algorithm. We find a significant improvement as
far as the statistical uncertainty is concerned at a similar numerical effort.
In order to efficiently train large neural networks we introduce the technique
of pre-training. It allows to train some neural networks using smaller system
sizes and then employing them as starting configurations for larger system
sizes. This is possible due to the recursive construction of our hierarchical
approach. Our results serve as a demonstration of the performance of the
hierarchical approach for systems exhibiting bimodal distributions.
Additionally, we provide estimates of the free energy and entropy in the
vicinity of the phase transition with statistical uncertainties of the order of
$10^{-7}$ for the former and $10^{-3}$ for the latter based on a statistics of
$10^6$ configurations.
( 2
min )
The approximation properties of infinitely wide shallow neural networks
heavily depend on the choice of the activation function. To understand this
influence, we study embeddings between Barron spaces with different activation
functions. These embeddings are proven by providing push-forward maps on the
measures $\mu$ used to represent functions $f$. An activation function of
particular interest is the rectified power unit ($\operatorname{RePU}$) given
by $\operatorname{RePU}_s(x)=\max(0,x)^s$. For many commonly used activation
functions, the well-known Taylor remainder theorem can be used to construct a
push-forward map, which allows us to prove the embedding of the associated
Barron space into a Barron space with a $\operatorname{RePU}$ as activation
function. Moreover, the Barron spaces associated with the
$\operatorname{RePU}_s$ have a hierarchical structure similar to the Sobolev
spaces $H^m$.
( 2
min )
We propose EB-TC$\varepsilon$, a novel sampling rule for $\varepsilon$-best
arm identification in stochastic bandits. It is the first instance of Top Two
algorithm analyzed for approximate best arm identification. EB-TC$\varepsilon$
is an *anytime* sampling rule that can therefore be employed without
modification for fixed confidence or fixed budget identification (without prior
knowledge of the budget). We provide three types of theoretical guarantees for
EB-TC$\varepsilon$. First, we prove bounds on its expected sample complexity in
the fixed confidence setting, notably showing its asymptotic optimality in
combination with an adaptive tuning of its exploration parameter. We complement
these findings with upper bounds on its probability of error at any time and
for any error parameter, which further yield upper bounds on its simple regret
at any time. Finally, we show through numerical simulations that
EB-TC$\varepsilon$ performs favorably compared to existing algorithms, in
different settings.
( 2
min )
The research explores the influence of preschool attendance (one year before
full-time school) on the development of children during their first year of
school. Using data collected by the Australian Early Development Census, the
findings show that areas with high proportions of preschool attendance tended
to have lower proportions of children with at least one developmental
vulnerability. Developmental vulnerablities include not being able to cope with
the school day (tired, hungry, low energy), unable to get along with others or
aggressive behaviour, trouble with reading/writing or numbers. These findings,
of course, vary by region. Using Data Analysis and Machine Learning, the
researchers were able to identify three distinct clusters within Queensland,
each characterised by different socio-demographic variables influencing the
relationship between preschool attendance and developmental vulnerability.
These analyses contribute to understanding regions with high vulnerability and
the potential need for tailored policies or investments
( 2
min )
Many two-sample network hypothesis testing methodologies operate under the
implicit assumption that the vertex correspondence across networks is a priori
known. In this paper, we consider the degradation of power in two-sample graph
hypothesis testing when there are misaligned/label-shuffled vertices across
networks. In the context of random dot product and stochastic block model
networks, we theoretically explore the power loss due to shuffling for a pair
of hypothesis tests based on Frobenius norm differences between estimated edge
probability matrices or between adjacency matrices. The loss in testing power
is further reinforced by numerous simulations and experiments, both in the
stochastic block model and in the random dot product graph model, where we
compare the power loss across multiple recently proposed tests in the
literature. Lastly, we demonstrate the impact that shuffling can have in
real-data testing in a pair of examples from neuroscience and from social
network analysis.
( 2
min )
We introduce Markov Neural Processes (MNPs), a new class of Stochastic
Processes (SPs) which are constructed by stacking sequences of neural
parameterised Markov transition operators in function space. We prove that
these Markov transition operators can preserve the exchangeability and
consistency of SPs. Therefore, the proposed iterative construction adds
substantial flexibility and expressivity to the original framework of Neural
Processes (NPs) without compromising consistency or adding restrictions. Our
experiments demonstrate clear advantages of MNPs over baseline models on a
variety of tasks.
( 2
min )
A crucial problem in reinforcement learning is learning the optimal policy.
We study this in tabular infinite-horizon discounted Markov decision processes
under the online setting. The existing algorithms either fail to achieve regret
optimality or have to incur a high memory and computational cost. In addition,
existing optimal algorithms all require a long burn-in time in order to achieve
optimal sample efficiency, i.e., their optimality is not guaranteed unless
sample size surpasses a high threshold. We address both open problems by
introducing a model-free algorithm that employs variance reduction and a novel
technique that switches the execution policy in a slow-yet-adaptive manner.
This is the first regret-optimal model-free algorithm in the discounted
setting, with the additional benefit of a low burn-in time.
( 2
min )
Text-to-image generation is a task in which a machine learning (ML) model generates an image from a textual description. The goal is to generate an image that closely matches the description, capturing the details and nuances of the text. This task is challenging because it requires the model to understand the semantics and syntax of […]
( 15
min )
AI Weirdness: the strange side of machine learning
( 2
min )
One of the most common applications of generative AI and large language models (LLMs) in an enterprise environment is answering questions based on the enterprise’s knowledge corpus. Amazon Lex provides the framework for building AI based chatbots. Pre-trained foundation models (FMs) perform well at natural language understanding (NLU) tasks such summarization, text generation and question […]
( 12
min )
Amazon Kendra is a highly accurate and intelligent search service that enables users to search unstructured and structured data using natural language processing (NLP) and advanced search algorithms. With Amazon Kendra, you can find relevant answers to your questions quickly, without sifting through documents. However, just enabling end-users to get the answers to their queries […]
( 10
min )
This post was co-authored by Brian Curry (Founder and Head of Products at OCX Cognition) and Sandhya MN (Data Science Lead at InfoGain) OCX Cognition is a San Francisco Bay Area-based startup, offering a commercial B2B software as a service (SaaS) product called Spectrum AI. Spectrum AI is a predictive (generative) CX analytics platform for […]
( 8
min )
Two years after he spoke at a conference detailing his ambitious vision for cooling tomorrow’s data centers, Ali Heydari and his team won a $5 million grant to go build it. It was the largest of 15 awards in May from the U.S. Department of Energy. The DoE program, called COOLERCHIPS, received more than 100 Read article >
( 6
min )
For about six years, AI has been an integral part of the artwork of Dominic Harris, a London-based digital artist who’s about to launch his biggest exhibition to date. “I use it for things like giving butterflies a natural sense of movement,” said Harris, whose typical canvas is an interactive computer display. Using a rack Read article >
( 6
min )
The machine-learning algorithm identified a compound that kills Acinetobacter baumannii, a bacterium that lurks in many hospital settings.
( 9
min )
It’s more important than ever for artificial intelligence to estimate how accurately it is explaining data.
( 8
min )
Data and its by-products dominate the world we live in. Smartphones and easy Internet access have increased this proliferation of data at a much higher rate than before. To make sense of this data and to use it for business advantage, companies analyze this huge amount of data to get insights. Such insights from text… Read More »Key benefits of using text visualizations for your business
The post Key benefits of using text visualizations for your business appeared first on Data Science Central.
( 22
min )
Cyber security experts face a tough challenge from the new type of quantum computers capable of easily breaking through security codes. Quantum computers, based on principles of quantum physics instead of standard electronic systems, are still nascent and do not have enough processing power to crack encryption keys. However, the experts at QDex Labs believe that the… Read More »Quantum resistant cryptography – bolstering cyber security against the threats posed by quantum computing
The post Quantum resistant cryptography – bolstering cyber security against the threats posed by quantum computing appeared first on Data Science Central.
( 19
min )
ChatGPT continues to revolutionize the way financial conversations are conducted, by providing its users with a fast and reliable tool for decision-making. The synergy between Bitcoin and ChatGPT is evident in how each technology enables the other to reach its full potential. Bitcoin provides an efficient payment system, while ChatGPT enhances conversational capabilities through natural… Read More »Exploring the Synergy between Bitcoin and ChatGPT: Empowering Financial Conversations
The post Exploring the Synergy between Bitcoin and ChatGPT: Empowering Financial Conversations appeared first on Data Science Central.
( 22
min )
There is no denying that Artificial Intelligence is revolutionizing the business landscape in almost every industry. With the advent of new possible applications and the ongoing process of improving existing ones, AI is opening up exciting opportunities for those ready to take them. One key trend in this industry is personalization and precision marketing, which… Read More »Personalization and precision marketing: Revenue streams in CPGs through AI
The post Personalization and precision marketing: Revenue streams in CPGs through AI appeared first on Data Science Central.
( 21
min )
Intelligent document processing (IDP) is a technology that automates the processing of high volumes of unstructured data, including text, images, and videos. IDP offers a significant improvement over manual methods and legacy optical character recognition (OCR) systems by addressing challenges such as cost, errors, low accuracy, and limited scalability, ultimately leading to better outcomes for […]
( 18
min )
In this three-part series, we present a solution that demonstrates how you can automate detecting document tampering and fraud at scale using AWS AI and machine learning (ML) services for a mortgage underwriting use case. This solution rides on a more significant global wave of increasing mortgage fraud, which is worsening as more people present […]
( 8
min )
Today we are excited to announce that you can now perform batch transforms with Amazon SageMaker JumpStart large language models (LLMs) for Text2Text Generation. Batch transforms are useful in situations where the responses don’t need to be real time and therefore you can do inference in batch for large datasets in bulk. For batch transform, […]
( 12
min )
The GeForce RTX 4060 Ti 8GB GPU is now available from top add-in card providers including ASUS, Colorful, Galax, GIGABYTE, INNO3D, MSI, Palit, PNY and ZOTAC, as well as from system integrators and builders worldwide.
( 7
min )
When we start learning Python, many times, we come across bad practices. In this article, you will learn the best practices to take your…
( 16
min )
Announcements TLADS and the Socratic Method: Bill Schmarzo’s Excellent Adventure Frequent Data Science Central contributor Bill Schmarzo has long touted the “Think Like a Data Scientist” methodology for business decisions. Bill notes that when leaders (and employees) “TLADS,” it provides a framework for value-based problem-solving and data-driven decision-making. By incorporating business context, stakeholder alignment and… Read More »DSC Weekly 23 May 2023 – TLADS and the Socratic Method: Bill Schmarzo’s Excellent Adventure
The post DSC Weekly 23 May 2023 – TLADS and the Socratic Method: Bill Schmarzo’s Excellent Adventure appeared first on Data Science Central.
( 19
min )
The healthcare industry relies heavily on accurate claims auditing to ensure proper reimbursement and financial stability. Claims auditors must determine the correct party, membership eligibility, contractual adherence, and fraud, waste, and abuse to accurately pay to prepay and postpay healthcare claims. This is a difficult task with many obstacles. Healthcare reimbursement and financial stability depend… Read More »AI-Assisted Claims Auditing: Uncovering Errors Leading to Boosted Financial Recovery
The post AI-Assisted Claims Auditing: Uncovering Errors Leading to Boosted Financial Recovery appeared first on Data Science Central.
( 22
min )
By Jess Warrington, General Manager, North America, CloudBlue They say eCommerce is the new normal, but beyond simple selling, it has ushered in the next evolution of B2B transactions. Digital marketplaces enable tech vendors to broaden their reach and expand their catalog of products and services, giving companies the ability to package multiple types of… Read More »How Tech Vendors Can Embrace the Digital Marketplace Reset – Tips on navigating the digital marketplace-as-a-service landscape
The post How Tech Vendors Can Embrace the Digital Marketplace Reset – Tips on navigating the digital marketplace-as-a-service landscape appeared first on Data Science Central.
( 21
min )
Most of us agree that search is broken. It has not changed much in terms of user experience over the last two decades. To make matters worse, due to the SEO/ad driven focus, the results from search are often preceded by advertising. Gen Z has realised this and are using TikTok and other platforms as… Read More »LLM results in search – Google search perspectives and generative AI in search
The post LLM results in search – Google search perspectives and generative AI in search appeared first on Data Science Central.
( 19
min )
FAIR Data Forecast interview with Todd Carter “Most video assets are hugely underperforming,” Todd Carter, CTO of Resolute Square, said in our Personal Knowledge Graph working group interview with him. “I know you all are practitioners used to indexable metadata, but that’s not what we have here.” Resolute Square (RS) is a Public Benefit Corporation… Read More »Boosting video “surface area” for discoverability with knowledge graphs
The post Boosting video “surface area” for discoverability with knowledge graphs appeared first on Data Science Central.
( 20
min )
Smile, you are being watched. Over the past few years, facial recognition technology has captivated the world with its awe and apprehension. Everyone in the tech world knows about it, but few of us know what happens behind the scenes. Similar to celebrity gossip, everyone knows what happens behind the scenes regarding the latest celebrities,… Read More »The Future of Facial Recognition: Promoting Responsible Deployment and Ethical Practices
The post The Future of Facial Recognition: Promoting Responsible Deployment and Ethical Practices appeared first on Data Science Central.
( 22
min )
Amazon Kendra is a highly accurate and simple-to-use intelligent search service powered by machine learning (ML). Amazon Kendra offers a suite of data source connectors to simplify the process of ingesting and indexing your content, wherever it resides. Valuable data in organizations is stored in both structured and unstructured repositories. An enterprise search solution should […]
( 12
min )
This is a guest blog post co-written with Vik Pant and Kyle Bassett from PwC. With organizations increasingly investing in machine learning (ML), ML adoption has become an integral part of business transformation strategies. A recent PwC CEO survey unveiled that 84% of Canadian CEOs agree that artificial intelligence (AI) will significantly change their business […]
( 8
min )
The seeds of a machine learning (ML) paradigm shift have existed for decades, but with the ready availability of virtually infinite compute capacity, a massive proliferation of data, and the rapid advancement of ML technologies, customers across industries are rapidly adopting and using ML technologies to transform their businesses. Just recently, generative AI applications have […]
( 13
min )
Generative AI — in the form of large language model (LLM) applications like ChatGPT, image generators such as Stable Diffusion and Adobe Firefly, and game rendering techniques like NVIDIA DLSS 3 Frame Generation — is rapidly ushering in a new era of computing for productivity, content creation, gaming and more. At the Microsoft Build developer Read article >
( 7
min )
Robotics hardware traditionally requires programmers to deploy it. READY Robotics wants to change that with its “no code” software aimed at people working in manufacturing who haven’t got programming skills. The Columbus, Ohio, startup is a spinout of robotics research from Johns Hopkins University. Kel Guerin was a PhD candidate there leading this research when Read article >
( 6
min )
It’s time to take out the space trash. In this episode of the NVIDIA AI Podcast, host Noah Kravitz dives into an illuminating conversation with Alex Fielding, co-founder and CEO of Privateer Space. Fielding is a tech industry veteran, having previously worked alongside Apple co-founder Steve Wozniak on several projects, and holds a deep expertise Read article >
( 4
min )
TL;DR: Text Prompt -> LLM -> Intermediate Representation (such as an image layout) -> Stable Diffusion -> Image.
Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, despite their impressive capabilities, diffusion models, such as Stable Diffusion, often struggle to accurately follow the prompts when spatial or common sense reasoning is required.
The following figure lists four scenarios in which Stable Diffusion falls short in generating images that accurately correspond to the given prompts, namely negation, numeracy, and attribute assignment, spatial relationships. In contrast, our method, LLM-grounded Diffusion (LMD), delivers much better prompt understanding in text-to-image gen…
( 3
min )
This machine-learning method could assist with robotic scene understanding, image editing, or online recommendation systems.
( 10
min )
Introduction:
( 11
min )
Generative AI is in the midst of a period of stunning growth. Increasingly capable foundation models are being released continuously, with large language models (LLMs) being one of the most visible model classes. LLMs are models composed of billions of parameters trained on extensive corpora of text, up to hundreds of billions or even a […]
( 17
min )
People agree: accelerated computing is energy-efficient computing. The National Energy Research Scientific Computing Center (NERSC), the U.S. Department of Energy’s lead facility for open science, measured results across four of its key high performance computing and AI applications. They clocked how fast the applications ran and how much energy they consumed on CPU-only and GPU-accelerated Read article >
( 5
min )
In this work, we explore Parameter-Efficient-Learning (PEL) techniques to
repurpose a General-Purpose-Speech (GSM) model for Arabic dialect
identification (ADI). Specifically, we investigate different setups to
incorporate trainable features into a multi-layer encoder-decoder GSM
formulation under frozen pre-trained settings. Our architecture includes
residual adapter and model reprogramming (input-prompting). We design a
token-level label mapping to condition the GSM for Arabic Dialect
Identification (ADI). This is challenging due to the high variation in
vocabulary and pronunciation among the numerous regional dialects. We achieve
new state-of-the-art accuracy on the ADI-17 dataset by vanilla fine-tuning. We
further reduce the training budgets with the PEL method, which performs within
1.86% accuracy to fine-tuning using only 2.5% of (extra) network trainable
parameters. Our study demonstrates how to identify Arabic dialects using a
small dataset and limited computation with open source code and pre-trained
models.
( 2
min )
Machine learning (ML) has revolutionized transportation systems, enabling
autonomous driving and smart traffic services. Federated learning (FL)
overcomes privacy constraints by training ML models in distributed systems,
exchanging model parameters instead of raw data. However, the dynamic states of
connected vehicles affect the network connection quality and influence the FL
performance. To tackle this challenge, we propose a contextual client selection
pipeline that uses Vehicle-to-Everything (V2X) messages to select clients based
on the predicted communication latency. The pipeline includes: (i) fusing V2X
messages, (ii) predicting future traffic topology, (iii) pre-clustering clients
based on local data distribution similarity, and (iv) selecting clients with
minimal latency for future model aggregation. Experiments show that our
pipeline outperforms baselines on various datasets, particularly in non-iid
settings.
( 2
min )
Deep ensembles achieved state-of-the-art results in classification and
out-of-distribution (OOD) detection; however, their effectiveness remains
limited due to the homogeneity of learned patterns within the ensemble. To
overcome this challenge, our study introduces a novel approach that promotes
diversity among ensemble members by leveraging saliency maps. By incorporating
saliency map diversification, our method outperforms conventional ensemble
techniques in multiple classification and OOD detection tasks, while also
improving calibration. Experiments on well-established OpenOOD benchmarks
highlight the potential of our method in practical applications.
( 2
min )
The recent emergence of Self-Supervised Learning (SSL) as a fundamental
paradigm for learning image representations has, and continues to, demonstrate
high empirical success in a variety of tasks. However, most SSL approaches fail
to learn embeddings that capture hierarchical semantic concepts that are
separable and interpretable. In this work, we aim to learn highly separable
semantic hierarchical representations by stacking Joint Embedding Architectures
(JEA) where higher-level JEAs are input with representations of lower-level
JEA. This results in a representation space that exhibits distinct
sub-categories of semantic concepts (e.g., model and colour of vehicles) in
higher-level JEAs. We empirically show that representations from stacked JEA
perform on a similar level as traditional JEA with comparative parameter counts
and visualise the representation spaces to validate the semantic hierarchies.
( 2
min )
Neural machine translation (NMT) has become the de-facto standard in
real-world machine translation applications. However, NMT models can
unpredictably produce severely pathological translations, known as
hallucinations, that seriously undermine user trust. It becomes thus crucial to
implement effective preventive strategies to guarantee their proper
functioning. In this paper, we address the problem of hallucination detection
in NMT by following a simple intuition: as hallucinations are detached from the
source content, they exhibit encoder-decoder attention patterns that are
statistically different from those of good quality translations. We frame this
problem with an optimal transport formulation and propose a fully unsupervised,
plug-in detector that can be used with any attention-based NMT model.
Experimental results show that our detector not only outperforms all previous
model-based detectors, but is also competitive with detectors that employ large
models trained on millions of samples.
( 2
min )
Parkinson's disease (PD) is a neurological disorder impacting a person's
speech. Among automatic PD assessment methods, deep learning models have gained
particular interest. Recently, the community has explored cross-pathology and
cross-language models which can improve diagnostic accuracy even further.
However, strict patient data privacy regulations largely prevent institutions
from sharing patient speech data with each other. In this paper, we employ
federated learning (FL) for PD detection using speech signals from 3 real-world
language corpora of German, Spanish, and Czech, each from a separate
institution. Our results indicate that the FL model outperforms all the local
models in terms of diagnostic accuracy, while not performing very differently
from the model based on centrally combined training sets, with the advantage of
not requiring any data sharing among collaborators. This will simplify
inter-institutional collaborations, resulting in enhancement of patient
outcomes.
( 2
min )
We present a machine learning approach for the aftershock forecasting of
Japanese earthquake catalogue from 2015 to 2019. Our method takes as sole input
the ground surface deformation as measured by Global Positioning System (GPS)
stations at the day of the mainshock, and processes it with a Convolutional
Neural Network (CNN), thus capturing the input's spatial correlations. Despite
the moderate amount of data the performance of this new approach is very
promising. The accuracy of the prediction heavily relies on the density of GPS
stations: the predictive power is lost when the mainshocks occur far from
measurement stations, as in offshore regions.
( 2
min )
We describe a proof-of-principle implementation of a system for drawing
melodies that abstracts away from a note-level input representation via melodic
contours. The aim is to allow users to express their musical intentions without
requiring prior knowledge of how notes fit together melodiously. Current
approaches to controllable melody generation often require users to choose
parameters that are static across a whole sequence, via buttons or sliders. In
contrast, our method allows users to quickly specify how parameters should
change over time by drawing a contour.
( 2
min )
In this paper, we investigate the complexity of feed-forward neural networks
by examining the concept of functional equivalence, which suggests that
different network parameterizations can lead to the same function. We utilize
the permutation invariance property to derive a novel covering number bound for
the class of feedforward neural networks, which reveals that the complexity of
a neural network can be reduced by exploiting this property. Furthermore, based
on the symmetric structure of parameter space, we demonstrate that an
appropriate strategy of random parameter initialization can increase the
probability of convergence for optimization. We found that overparameterized
networks tend to be easier to train in the sense that increasing the width of
neural networks leads to a vanishing volume of the effective parameter space.
Our findings offer new insights into overparameterization and have significant
implications for understanding generalization and optimization in deep
learning.
( 2
min )
Identifying molecules that exhibit some pre-specified properties is a
difficult problem to solve. In the last few years, deep generative models have
been used for molecule generation. Deep Graph Variational Autoencoders are
among the most powerful machine learning tools with which it is possible to
address this problem. However, existing methods struggle in capturing the true
data distribution and tend to be computationally expensive. In this work, we
propose RGCVAE, an efficient and effective Graph Variational Autoencoder based
on: (i) an encoding network exploiting a new powerful Relational Graph
Isomorphism Network; (ii) a novel probabilistic decoding component. Compared to
several state-of-the-art VAE methods on two widely adopted datasets, RGCVAE
shows state-of-the-art molecule generation performance while being
significantly faster to train.
( 2
min )
This work presents a comprehensive analysis to regularize the Soft
Actor-Critic (SAC) algorithm with automatic temperature adjustment. The the
policy evaluation, the policy improvement and the temperature adjustment are
reformulated, addressing certain modification and enhancing the clarity of the
original theory in a more explicit manner.
( 2
min )
This paper presents a community-centered study of cultural limitations of
text-to-image (T2I) models in the South Asian context. We theorize these
failures using scholarship on dominant media regimes of representations and
locate them within participants' reporting of their existing social
marginalizations. We thus show how generative AI can reproduce an outsiders
gaze for viewing South Asian cultures, shaped by global and regional power
inequities. By centering communities as experts and soliciting their
perspectives on T2I limitations, our study adds rich nuance into existing
evaluative frameworks and deepens our understanding of the culturally-specific
ways AI technologies can fail in non-Western and Global South settings. We
distill lessons for responsible development of T2I models, recommending
concrete pathways forward that can allow for recognition of structural
inequalities.
( 2
min )
The recent rapid progress in pre-training Large Language Models has relied on
using self-supervised language modeling objectives like next token prediction
or span corruption. On the other hand, Machine Translation Systems are mostly
trained using cross-lingual supervision that requires aligned data between
source and target languages. We demonstrate that pre-training Large Language
Models on a mixture of a self-supervised Language Modeling objective and the
supervised Machine Translation objective, therefore including cross-lingual
parallel data during pre-training, yields models with better in-context
learning abilities. As pre-training is a very resource-intensive process and a
grid search on the best mixing ratio between the two objectives is
prohibitively expensive, we propose a simple yet effective strategy to learn it
during pre-training.
( 2
min )
In this paper, we study the statistical efficiency of Reinforcement Learning
in Mean-Field Control (MFC) and Mean-Field Game (MFG) with general function
approximation. We introduce a new concept called Mean-Field Model-Based Eluder
Dimension (MBED), which subsumes a rich family of Mean-Field RL problems.
Additionally, we propose algorithms based on Optimistic Maximal Likelihood
Estimation, which can return an $\epsilon$-optimal policy for MFC or an
$\epsilon$-Nash Equilibrium policy for MFG, with sample complexity polynomial
w.r.t. relevant parameters and independent of the number of states, actions and
the number of agents. Notably, our results only require a mild assumption of
Lipschitz continuity on transition dynamics and avoid strong structural
assumptions in previous work. Finally, in the tabular setting, given the access
to a generative model, we establish an exponential lower bound for MFC setting,
while providing a novel sample-efficient model elimination algorithm to
approximate equilibrium in MFG setting. Our results reveal a fundamental
separation between RL for single-agent, MFC, and MFG from the sample efficiency
perspective.
( 2
min )
A Markov network characterizes the conditional independence structure, or
Markov property, among a set of random variables. Existing work focuses on
specific families of distributions (e.g., exponential families) and/or certain
structures of graphs, and most of them can only handle variables of a single
data type (continuous or discrete). In this work, we characterize the
conditional independence structure in general distributions for all data types
(i.e., continuous, discrete, and mixed-type) with a Generalized Precision
Matrix (GPM). Besides, we also allow general functional relations among
variables, thus giving rise to a Markov network structure learning algorithm in
one of the most general settings. To deal with the computational challenge of
the problem, especially for large graphs, we unify all cases under the same
umbrella of a regularized score matching framework. We validate the theoretical
results and demonstrate the scalability empirically in various settings.
( 2
min )
We consider the community recovery problem on a multilayer variant of the
hypergraph stochastic block model (HSBM). Each layer is associated with an
independent realization of a d-uniform HSBM on N vertices. Given the similarity
matrix containing the aggregated number of hyperedges incident to each pair of
vertices, the goal is to obtain a partition of the N vertices into disjoint
communities. In this work, we investigate a semidefinite programming (SDP)
approach and obtain information-theoretic conditions on the model parameters
that guarantee exact recovery both in the assortative and the disassortative
cases.
( 2
min )
In this paper, we study the statistical efficiency of Reinforcement Learning
in Mean-Field Control (MFC) and Mean-Field Game (MFG) with general function
approximation. We introduce a new concept called Mean-Field Model-Based Eluder
Dimension (MBED), which subsumes a rich family of Mean-Field RL problems.
Additionally, we propose algorithms based on Optimistic Maximal Likelihood
Estimation, which can return an $\epsilon$-optimal policy for MFC or an
$\epsilon$-Nash Equilibrium policy for MFG, with sample complexity polynomial
w.r.t. relevant parameters and independent of the number of states, actions and
the number of agents. Notably, our results only require a mild assumption of
Lipschitz continuity on transition dynamics and avoid strong structural
assumptions in previous work. Finally, in the tabular setting, given the access
to a generative model, we establish an exponential lower bound for MFC setting,
while providing a novel sample-efficient model elimination algorithm to
approximate equilibrium in MFG setting. Our results reveal a fundamental
separation between RL for single-agent, MFC, and MFG from the sample efficiency
perspective.
( 2
min )
A fundamental problem of causal discovery is cause-effect inference, learning
the correct causal direction between two random variables. Significant progress
has been made through modelling the effect as a function of its cause and a
noise term, which allows us to leverage assumptions about the generating
function class. The recently introduced heteroscedastic location-scale noise
functional models (LSNMs) combine expressive power with identifiability
guarantees. LSNM model selection based on maximizing likelihood achieves
state-of-the-art accuracy, when the noise distributions are correctly
specified. However, through an extensive empirical evaluation, we demonstrate
that the accuracy deteriorates sharply when the form of the noise distribution
is misspecified by the user. Our analysis shows that the failure occurs mainly
when the conditional variance in the anti-causal direction is smaller than that
in the causal direction. As an alternative, we find that causal model selection
through residual independence testing is much more robust to noise
misspecification and misleading conditional variance.
( 2
min )
We propose a new multimodal variational autoencoder that enables to generate
from the joint distribution and conditionally to any number of complex
modalities. The unimodal posteriors are conditioned on the Deep Canonical
Correlation Analysis embeddings which preserve the shared information across
modalities leading to more coherent cross-modal generations. Furthermore, we
use Normalizing Flows to enrich the unimodal posteriors and achieve more
diverse data generation. Finally, we propose to use a Product of Experts for
inferring one modality from several others which makes the model scalable to
any number of modalities. We demonstrate that our method improves likelihood
estimates, diversity of the generations and in particular coherence metrics in
the conditional generations on several datasets.
( 2
min )
Senior Ananya Gurumurthy adds her musical talents to her math and computer science studies to advocate using data for social change.
( 9
min )
In this paper, we present a robust incremental learning model for regression
tasks on temporal tabular datasets. Using commonly available tabular and
time-series prediction models as building blocks, a machine-learning model is
built incrementally to adapt to distributional shifts in data. Using the
concept of self-similarity, the model uses only two basic building blocks of
machine learning models, gradient boosting decision trees and neural networks
to build models for any required complexity. The model is efficient as no
specialised neural architectures are used and each model building block can be
independently trained in parallel. The model is demonstrated to have robust
performances under adverse situations such as regime changes, fat-tailed
distributions and low signal-to-noise ratios. Model robustness are studied
under different hyper-parameters and complexities.
( 2
min )
If you are curious about the ethical considerations and debates surrounding AI-generated art, then this blog post is for you. I will be…
( 18
min )
The transportation and logistics industry has undergone a massive change with the introduction of artificial intelligence. After the…
( 13
min )
The banking sector is one of the most significant industries and is heavily dependent on technology to meet customer needs, build customer…
( 12
min )
Vision loss comes in various forms. For some, it’s from birth, for others, it’s a slow descent over time which comes with many expiration dates: The day you can’t see pictures, recognize yourself, or loved ones faces or even read your mail. In our previous blogpost Enable the Visually Impaired to Hear Documents using Amazon […]
( 9
min )
We introduce HATELEXICON, a lexicon of slurs and targets of hate speech for
the countries of Brazil, Germany, India and Kenya, to aid training and
interpretability of models. We demonstrate how our lexicon can be used to
interpret model predictions, showing that models developed to classify extreme
speech rely heavily on target words when making predictions. Further, we
propose a method to aid shot selection for training in low-resource settings
via HATELEXICON. In few-shot learning, the selection of shots is of paramount
importance to model performance. In our work, we simulate a few-shot setting
for German and Hindi, using HASOC data for training and the Multilingual
HateCheck (MHC) as a benchmark. We show that selecting shots based on our
lexicon leads to models performing better on MHC than models trained on shots
sampled randomly. Thus, when given only a few training examples, using our
lexicon to select shots containing more sociocultural information leads to
better few-shot performance.
( 2
min )
Entropy measures are effective features for time series classification
problems. Traditional entropy measures, such as Shannon entropy, use
probability distribution function. However, for the effective separation of
time series, new entropy estimation methods are required to characterize the
chaotic dynamic of the system. Our concept of Neural Network Entropy (NNetEn)
is based on the classification of special datasets in relation to the entropy
of the time series recorded in the reservoir of the neural network. NNetEn
estimates the chaotic dynamics of time series in an original way and does not
take into account probability distribution functions. We propose two new
classification metrics: R2 Efficiency and Pearson Efficiency. The efficiency of
NNetEn is verified on separation of two chaotic time series of sine mapping
using dispersion analysis. For two close dynamic time series (r = 1.1918 and r
= 1.2243), the F-ratio has reached the value of 124 and reflects high
efficiency of the introduced method in classification problems. The
electroenceph-alography signal classification for healthy persons and patients
with Alzheimer disease illustrates the practical application of the NNetEn
features. Our computations demonstrate the synergistic effect of increasing
classification accuracy when applying traditional entropy measures and the
NNetEn concept conjointly. An implementation of the algorithms in Python is
presented.
( 3
min )
We study the gradients of a maxout network with respect to inputs and
parameters and obtain bounds for the moments depending on the architecture and
the parameter distribution. We observe that the distribution of the
input-output Jacobian depends on the input, which complicates a stable
parameter initialization. Based on the moments of the gradients, we formulate
parameter initialization strategies that avoid vanishing and exploding
gradients in wide networks. Experiments with deep fully-connected and
convolutional networks show that this strategy improves SGD and Adam training
of deep maxout networks. In addition, we obtain refined bounds on the expected
number of linear regions, results on the expected curve length distortion, and
results on the NTK.
( 2
min )
We prove Wasserstein inverse reinforcement learning enables the learner's
reward values to imitate the expert's reward values in a finite iteration for
multi-objective optimizations. Moreover, we prove Wasserstein inverse
reinforcement learning enables the learner's optimal solutions to imitate the
expert's optimal solutions for multi-objective optimizations with lexicographic
order.
( 2
min )
A solar sail is one of the most promising space exploration system because of
its theoretically infinite specific impulse using solar radiation pressure
(SRP). Recently, some researchers proposed "transformable spacecrafts" that can
actively reconfigure their body configurations with actuatable joints. The
transformable spacecrafts are expected to greatly enhance orbit and attitude
control capability due to its high redundancy in control degree of freedom if
they are used like solar sails. However, its large number of input poses
difficulties in control, and therefore, previous researchers imposed strong
constraints to limit its potential control capabilities. This paper addresses
novel attitude control techniques for the transformable spacecrafts under SRP.
The authors have constructed two proposed methods; one of those is a joint
angle optimization to acquire arbitrary SRP force and torque, and the other is
a momentum damping control driven by joint angle actuation. Our proposed
methods are formulated in general forms and applicable to any transformable
spacecraft that has front faces that can dominantly receive SRP on each body.
Validity of the proposed methods are confirmed by numerical simulations. This
paper contributes to making most of the high control redundancy of
transformable spacecrafts without consuming any expendable propellants, which
is expected to greatly enhance orbit and attitude control capability.
( 3
min )
Although deep neural network (DNN)-based speech enhancement (SE) methods
outperform the previous non-DNN-based ones, they often degrade the perceptual
quality of generated outputs. To tackle this problem, we introduce a DNN-based
generative refiner, Diffiner, aiming to improve perceptual speech quality
pre-processed by an SE method. We train a diffusion-based generative model by
utilizing a dataset consisting of clean speech only. Then, our refiner
effectively mixes clean parts newly generated via denoising diffusion
restoration into the degraded and distorted parts caused by a preceding SE
method, resulting in refined speech. Once our refiner is trained on a set of
clean speech, it can be applied to various SE methods without additional
training specialized for each SE module. Therefore, our refiner can be a
versatile post-processing module w.r.t. SE methods and has high potential in
terms of modularity. Experimental results show that our method improved
perceptual speech quality regardless of the preceding SE methods used.
( 2
min )
Soft actor-critic is a successful successor over soft Q-learning. While lived
under maximum entropy framework, their relationship is still unclear. In this
paper, we prove that in the limit they converge to the same solution. This is
appealing since it translates the optimization from an arduous to an easier
way. The same justification can also be applied to other regularizers such as
KL divergence.
( 2
min )
Contemporary face detection algorithms have to deal with many challenges such
as variations in pose, illumination, and scale. A subclass of the face
detection problem that has recently gained increasing attention is occluded
face detection, or more specifically, the detection of masked faces. Three
years on since the advent of the COVID-19 pandemic, there is still a complete
lack of evidence regarding how well existing face detection algorithms perform
on masked faces. This article first offers a brief review of state-of-the-art
face detectors and detectors made for the masked face problem, along with a
review of the existing masked face datasets. We evaluate and compare the
performances of a well-representative set of face detectors at masked face
detection and conclude with a discussion on the possible contributing factors
to their performance.
( 2
min )
Large Language Models (LLMs) have made remarkable strides in natural language
processing, but their expanding size poses challenges in terms of computational
expense and inefficiency. Conversely, Small Language Models (SLMs) are known
for their efficiency but often encounter difficulties in tasks with limited
capacity and training data, particularly in domain-specific scenarios. In this
paper, we introduce Dr. LLaMA, a method that improves SLMs in the medical
domain through generative data augmentation utilizing LLMs. The objective is to
develop more efficient and capable models tailored for specialized
applications. Our preliminary results on the PubMedQA dataset demonstrate that
LLMs effectively refine and diversify existing question-answer pairs, leading
to improved performance of a significantly smaller model after fine-tuning. The
best SLM surpasses few-shot GPT-4 with under 1.6 billion parameters on the
PubMedQA. Our code and generated data are publicly available to facilitate
further explorations.
( 2
min )
We propose the first black-box targeted attack against online deep
reinforcement learning through reward poisoning during training time. Our
attack is applicable to general environments with unknown dynamics learned by
unknown algorithms and requires limited attack budgets and computational
resources. We leverage a general framework and find conditions to ensure
efficient attack under a general assumption of the learning algorithms. We show
that our attack is optimal in our framework under the conditions. We
experimentally verify that with limited budgets, our attack efficiently leads
the learning agent to various target policies under a diverse set of popular
DRL environments and state-of-the-art learners.
( 2
min )
Vector embeddings have been successfully applied in several domains to obtain
effective representations of non-numeric data which can then be used in various
downstream tasks. We present a novel application of vector embeddings in
professional road cycling by demonstrating a method to learn representations
for riders and races based on historical results. We use unsupervised learning
techniques to validate that the resultant embeddings capture interesting
features of riders and races. These embeddings could be used for downstream
prediction tasks such as early talent identification and race outcome
prediction.
( 2
min )
We investigate a primal-dual (PD) method for the saddle point problem (SPP)
that uses a linear approximation of the primal function instead of the standard
proximal step, resulting in a linearized PD (LPD) method. For convex-strongly
concave SPP, we observe that the LPD method has a suboptimal dependence on the
Lipschitz constant of the primal function. To fix this issue, we combine
features of Accelerated Gradient Descent with the LPD method resulting in a
single-loop Accelerated Linearized Primal-Dual (ALPD) method. ALPD method
achieves the optimal gradient complexity when the SPP has a semi-linear
coupling function. We also present an inexact ALPD method for SPPs with a
general nonlinear coupling function that maintains the optimal gradient
evaluations of the primal parts and significantly improves the gradient
evaluations of the coupling term compared to the ALPD method. We verify our
findings with numerical experiments.
( 2
min )
Uncertainty-quantification methods are applied to estimate the confidence of
deep-neural-networks classifiers over their predictions. However, most widely
used methods are known to be overconfident. We address this problem by
developing an algorithm that exploits the latent-space representation of data
points fed into the network, to assess the accuracy of their prediction. Using
the latent-space representation generated by the fraction of training set that
the network classifies correctly, we build a statistical model that is able to
capture the likelihood of a given prediction. We show on a synthetic dataset
that commonly used methods are mostly overconfident. Overconfidence occurs also
for predictions made on data points that are outside the distribution that
generated the training data. In contrast, our method can detect such
out-of-distribution data points as inaccurately predicted, thus aiding in the
automatic detection of outliers.
( 2
min )
We address the problem of denoising data from a Gaussian mixture using a
two-layer non-linear autoencoder with tied weights and a skip connection. We
consider the high-dimensional limit where the number of training samples and
the input dimension jointly tend to infinity while the number of hidden units
remains bounded. We provide closed-form expressions for the denoising
mean-squared test error. Building on this result, we quantitatively
characterize the advantage of the considered architecture over the autoencoder
without the skip connection that relates closely to principal component
analysis. We further show that our results accurately capture the learning
curves on a range of real data sets.
( 2
min )
There have thousands of crimes are happening daily all around. But people
keep statistics only few of them, therefore crime rates are increasing day by
day. The reason behind can be less concern or less statistics of previous
crimes. It is much more important to observe the previous crime statistics for
general people to make their outing decision and police for catching the
criminals are taking steps to restrain the crimes and tourists to make their
travelling decision. National institute of justice releases crime survey data
for the country, but does not offer crime statistics up to Union or Thana
level. Considering all of these cases we have come up with an approach which
can give an approximation to people about the safety of a specific location
with crime ranking of different areas locating the crimes on a map including a
future crime occurrence prediction mechanism. Our approach relies on different
online Bangla newspapers for crawling the crime data, stemming and keyword
extraction, location finding algorithm, cosine similarity, naive Bayes
classifier, and a custom crime prediction model
( 2
min )
We investigate how sparse neural activity affects the generalization
performance of a deep Bayesian neural network at the large width limit. To this
end, we derive a neural network Gaussian Process (NNGP) kernel with rectified
linear unit (ReLU) activation and a predetermined fraction of active neurons.
Using the NNGP kernel, we observe that the sparser networks outperform the
non-sparse networks at shallow depths on a variety of datasets. We validate
this observation by extending the existing theory on the generalization error
of kernel-ridge regression.
( 2
min )
Dynamic control flow is an important technique often used to design
expressive and efficient deep learning computations for applications such as
text parsing, machine translation, exiting early out of deep models and so on.
However, the resulting control flow divergence makes batching, an important
performance optimization, difficult to perform manually. In this paper, we
present ACRoBat, a framework that enables efficient automatic batching for
dynamic deep learning computations by performing hybrid static+dynamic compiler
optimizations and end-to-end tensor code generation. ACRoBat performs up to
8.5X better than DyNet, a state-of-the-art framework for automatic batching, on
an Nvidia GeForce RTX 3070 GPU.
( 2
min )
The core purpose of developing artificial neural networks was to mimic the
functionalities of biological neural networks. However, unlike biological
neural networks, traditional artificial neural networks are often structured
hierarchically, which can impede the flow of information between neurons as the
neurons in the same layer have no connections between them. Hence, we propose a
more robust model of artificial neural networks where the hidden neurons,
residing in the same hidden layer, are interconnected, enabling the neurons to
learn complex patterns and speeding up the convergence rate. With the
experimental study of our proposed model as fully connected layers in shallow
and deep networks, we demonstrate that the model results in a significant
increase in convergence rate.
( 2
min )
Most phenomena related to biomedical tasks are inherently complex, and in
many cases, are expressed as signals on biomedical Knowledge Graphs (KGs). In
this work, we introduce the use of a new representation framework, the Prime
Adjacency Matrix (PAM) for biomedical KGs, which allows for very efficient
network analysis. PAM utilizes prime numbers to enable representing the whole
KG with a single adjacency matrix and the fast computation of multiple
properties of the network. We illustrate the applicability of the framework in
the biomedical domain by working on different biomedical knowledge graphs and
by providing two case studies: one on drug-repurposing for COVID-19 and one on
important metapath extraction. We show that we achieve better results than the
original proposed workflows, using very simple methods that require no
training, in considerably less time.
( 2
min )
Off-Policy reinforcement learning has been a driving force for the
state-of-the-art conversational AIs leading to more natural humanagent
interactions and improving the user satisfaction for goal-oriented agents.
However, in large-scale commercial settings, it is often challenging to balance
between policy improvements and experience continuity on the broad spectrum of
applications handled by such system. In the literature, off-policy evaluation
and guard-railing on aggregate statistics has been commonly used to address
this problem. In this paper, we propose a method for curating and leveraging
high-precision samples sourced from historical regression incident reports to
validate, safe-guard, and improve policies prior to the online deployment. We
conducted extensive experiments using data from a real-world conversational
system and actual regression incidents. The proposed method is currently
deployed in our production system to protect customers against broken
experiences and enable long-term policy improvements.
( 2
min )
We prove a universal approximation property (UAP) for a class of ODENet and a
class of ResNet, which are simplified mathematical models for deep learning
systems with skip connections. The UAP can be stated as follows. Let $n$ and
$m$ be the dimension of input and output data, and assume $m\leq n$. Then we
show that ODENet of width $n+m$ with any non-polynomial continuous activation
function can approximate any continuous function on a compact subset on
$\mathbb{R}^n$. We also show that ResNet has the same property as the depth
tends to infinity. Furthermore, we derive the gradient of a loss function
explicitly with respect to a certain tuning variable. We use this to construct
a learning algorithm for ODENet. To demonstrate the usefulness of this
algorithm, we apply it to a regression problem, a binary classification, and a
multinomial classification in MNIST.
( 2
min )
We study the gradients of a maxout network with respect to inputs and
parameters and obtain bounds for the moments depending on the architecture and
the parameter distribution. We observe that the distribution of the
input-output Jacobian depends on the input, which complicates a stable
parameter initialization. Based on the moments of the gradients, we formulate
parameter initialization strategies that avoid vanishing and exploding
gradients in wide networks. Experiments with deep fully-connected and
convolutional networks show that this strategy improves SGD and Adam training
of deep maxout networks. In addition, we obtain refined bounds on the expected
number of linear regions, results on the expected curve length distortion, and
results on the NTK.
( 2
min )
Unifying semi-supervised learning (SSL) and open-set recognition into a
single learning policy would facilitate the development of cost-efficient and
application-grade classifiers. However, previous attempts do not clarify the
difference between unobserved novel categories (those only seen during testing)
and observed novel categories (those present in unlabelled training data). This
study introduces Open-Set Learning with Augmented Category by Exploiting
Unlabelled Data (Open-LACU), the first policy that generalises between both
novel category types. We adapt the state-of-the-art OSR method of Margin
Generative Adversarial Networks (Margin-GANs) into several Open-LACU
configurations, setting the benchmarks for Open-LACU and offering unique
insights into novelty detection using Margin-GANs. Finally, we highlight the
significance of the Open-LACU policy by discussing the applications of semantic
segmentation in remote sensing, object detection in radiology and disease
identification through cough analysis. These applications include observed and
unobserved novel categories, making Open-LACU essential for training
classifiers in these big data domains.
( 2
min )
This paper presents a general methodology for deriving information-theoretic
generalization bounds for learning algorithms. The main technical tool is a
probabilistic decorrelation lemma based on a change of measure and a relaxation
of Young's inequality in $L_{\psi_p}$ Orlicz spaces. Using the decorrelation
lemma in combination with other techniques, such as symmetrization, couplings,
and chaining in the space of probability measures, we obtain new upper bounds
on the generalization error, both in expectation and in high probability, and
recover as special cases many of the existing generalization bounds, including
the ones based on mutual information, conditional mutual information,
stochastic chaining, and PAC-Bayes inequalities. In addition, the
Fernique-Talagrand upper bound on the expected supremum of a subgaussian
process emerges as a special case.
( 2
min )
Considering the case where the response variable is a categorical variable
and the predictor is a random function, two novel functional sufficient
dimensional reduction (FSDR) methods are proposed based on mutual information
and square loss mutual information. Compared to the classical FSDR methods,
such as functional sliced inverse regression and functional sliced average
variance estimation, the proposed methods are appealing because they are
capable of estimating multiple effective dimension reduction directions in the
case of a relatively small number of categories, especially for the binary
response. Moreover, the proposed methods do not require the restrictive linear
conditional mean assumption and the constant covariance assumption. They avoid
the inverse problem of the covariance operator which is often encountered in
the functional sufficient dimension reduction. The functional principal
component analysis with truncation be used as a regularization mechanism. Under
some mild conditions, the statistical consistency of the proposed methods is
established. It is demonstrated that the two methods are competitive compared
with some existing FSDR methods by simulations and real data analyses.
( 2
min )
We focus on the task of learning a single index model $\sigma(w^\star \cdot
x)$ with respect to the isotropic Gaussian distribution in $d$ dimensions.
Prior work has shown that the sample complexity of learning $w^\star$ is
governed by the information exponent $k^\star$ of the link function $\sigma$,
which is defined as the index of the first nonzero Hermite coefficient of
$\sigma$. Ben Arous et al. (2021) showed that $n \gtrsim d^{k^\star-1}$ samples
suffice for learning $w^\star$ and that this is tight for online SGD. However,
the CSQ lower bound for gradient based methods only shows that $n \gtrsim
d^{k^\star/2}$ samples are necessary. In this work, we close the gap between
the upper and lower bounds by showing that online SGD on a smoothed loss learns
$w^\star$ with $n \gtrsim d^{k^\star/2}$ samples. We also draw connections to
statistical analyses of tensor PCA and to the implicit regularization effects
of minibatch SGD on empirical losses.
( 2
min )
We address the problem of denoising data from a Gaussian mixture using a
two-layer non-linear autoencoder with tied weights and a skip connection. We
consider the high-dimensional limit where the number of training samples and
the input dimension jointly tend to infinity while the number of hidden units
remains bounded. We provide closed-form expressions for the denoising
mean-squared test error. Building on this result, we quantitatively
characterize the advantage of the considered architecture over the autoencoder
without the skip connection that relates closely to principal component
analysis. We further show that our results accurately capture the learning
curves on a range of real data sets.
( 2
min )
Following up on a previous analysis of graph embeddings, we generalize and
expand some results to the general setting of vector symbolic architectures
(VSA) and hyperdimensional computing (HDC). Importantly, we explore the
mathematical relationship between superposition, orthogonality, and tensor
product. We establish the tensor product representation as the central
representation, with a suite of unique properties. These include it being the
most general and expressive representation, as well as being the most
compressed representation that has errorrless unbinding and detection.
( 2
min )
If you’re interested in stateful stream processing and the capabilities it provides, you may have heard of Apache Flink®. It’s well-known for its ability to perform stateful stream processing, but for beginners, it can be a daunting task to get started. So here, we’ll explore the basics of Apache Flink by showing you how to… Read More »Getting Started with Apache Flink: First steps to Stateful Stream Processing
The post Getting Started with Apache Flink: First steps to Stateful Stream Processing appeared first on Data Science Central.
( 22
min )
Amazon Kendra is a highly accurate and simple-to-use intelligent search service powered by machine learning (ML). Amazon Kendra offers a suite of data source connectors to simplify the process of ingesting and indexing your content, wherever it resides. Valuable data in organizations is stored in both structured and unstructured repositories. Amazon Kendra can pull together […]
( 9
min )
This research paper was accepted by the eighth ACM/IEEE Conference on Internet of Things Design and Implementation (IoTDI), which is a premier venue on IoT. The paper describes a framework that leverages cloud resources to execute large deep neural network (DNN) models with higher accuracy to improve the accuracy of models running on edge devices. The […]
The post REACT — A synergistic cloud-edge fusion architecture appeared first on Microsoft Research.
( 12
min )
Microsoft Editor provides AI-powered writing assistance to millions of users around the world. One of its features that writers of all levels and domains rely on is the grammar checker, which detects grammar errors in a user’s writing and offers suggested corrections and explanations of the detected errors. The technology behind grammar checker has evolved […]
The post Achieving Zero-COGS with Microsoft Editor Neural Grammar Checker appeared first on Microsoft Research.
( 17
min )
Scientific researchers need massive computational resources that can support exploration wherever it happens. Whether they’re conducting groundbreaking pharmaceutical research, exploring alternative energy sources or discovering new ways to prevent financial fraud, accessible state-of-the-art AI computing resources are key to driving innovation. This new model of computing can solve the challenges of generative AI and power Read article >
( 5
min )
The GeForce RTX 4060 family will be available starting next week, bringing massive creator benefits to the popular 60-class GPUs.
( 9
min )
The ChatGPT app syncs your conversations, supports voice input, and brings our latest model improvements to your fingertips.
( 1
min )
With the artificial intelligence conversation now mainstream, the 2023 MIT-MGB AI Cures conference saw attendance double from previous years.
( 8
min )
AWS delivers services that meet customers’ artificial intelligence (AI) and machine learning (ML) needs with services ranging from custom hardware like AWS Trainium and AWS Inferentia to generative AI foundation models (FMs) on Amazon Bedrock. In February 2022, AWS and Hugging Face announced a collaboration to make generative AI more accessible and cost efficient. Generative […]
( 7
min )
This post is co-written with Thatcher Thornberry from bpx energy. Facies classification is the process of segmenting lithologic formations from geologic data at the wellbore location. During drilling, wireline logs are obtained, which have depth-dependent geologic information. Geologists are deployed to analyze this log data and determine depth ranges for potential facies of interest from […]
( 11
min )
An update to the Omniverse Connector for Adobe Substance 3D Painter will save 3D creators across industries significant time and effort.
( 6
min )
In my last project, I had to come up with some code and algorithm to solve an interesting problem. I realized that it could lead to some off-the-beaten-path job interview question. The problem is a fundamental one. The level ranges from elementary school to one of the most difficult unsolved problems of all times, depending… Read More »An Intriguing Job Interview Question for AI/ML Professionals
The post An Intriguing Job Interview Question for AI/ML Professionals appeared first on Data Science Central.
( 21
min )
NASA's Kepler Space Telescope has been instrumental in the task of finding
the presence of exoplanets in our galaxy. This search has been supported by
computational data analysis to identify exoplanets from the signals received by
the Kepler telescope. In this paper, we consider building upon some existing
work on exoplanet identification using residual networks for the data of the
Kepler space telescope and its extended mission K2. This paper aims to explore
how deep learning algorithms can help in classifying the presence of exoplanets
with less amount of data in one case and a more extensive variety of data in
another. In addition to the standard CNN-based method, we propose a Siamese
architecture that is particularly useful in addressing classification in a
low-data scenario. The CNN and ResNet algorithms achieved an average accuracy
of 68% for three classes and 86% for two-class classification. However, for
both the three and two classes, the Siamese algorithm achieved 99% accuracy.
( 2
min )
D-Adaptation is an approach to automatically setting the learning rate which
asymptotically achieves the optimal rate of convergence for minimizing convex
Lipschitz functions, with no back-tracking or line searches, and no additional
function value or gradient evaluations per step. Our approach is the first
hyper-parameter free method for this class without additional multiplicative
log factors in the convergence rate. We present extensive experiments for SGD
and Adam variants of our method, where the method automatically matches
hand-tuned learning rates across more than a dozen diverse machine learning
problems, including large-scale vision and language problems.
An open-source implementation is available.
( 2
min )
Deep Learning (DL) has the potential to optimize machine learning in both the
scientific and clinical communities. However, greater expertise is required to
develop DL algorithms, and the variability of implementations hinders their
reproducibility, translation, and deployment. Here we present the
community-driven Generally Nuanced Deep Learning Framework (GaNDLF), with the
goal of lowering these barriers. GaNDLF makes the mechanism of DL development,
training, and inference more stable, reproducible, interpretable, and scalable,
without requiring an extensive technical background. GaNDLF aims to provide an
end-to-end solution for all DL-related tasks in computational precision
medicine. We demonstrate the ability of GaNDLF to analyze both radiology and
histology images, with built-in support for k-fold cross-validation, data
augmentation, multiple modalities and output classes. Our quantitative
performance evaluation on numerous use cases, anatomies, and computational
tasks supports GaNDLF as a robust application framework for deployment in
clinical workflows.
( 3
min )
Over-approximating the reachable sets of dynamical systems is a fundamental
problem in safety verification and robust control synthesis. The representation
of these sets is a key factor that affects the computational complexity and the
approximation error. In this paper, we develop a new approach for
over-approximating the reachable sets of neural network dynamical systems using
adaptive template polytopes. We use the singular value decomposition of linear
layers along with the shape of the activation functions to adapt the geometry
of the polytopes at each time step to the geometry of the true reachable sets.
We then propose a branch-and-bound method to compute accurate
over-approximations of the reachable sets by the inferred templates. We
illustrate the utility of the proposed approach in the reachability analysis of
linear systems driven by neural network controllers.
( 2
min )
A common pipeline in learning-based control is to iteratively estimate a
model of system dynamics, and apply a trajectory optimization algorithm -
e.g.~$\mathtt{iLQR}$ - on the learned model to minimize a target cost. This
paper conducts a rigorous analysis of a simplified variant of this strategy for
general nonlinear systems. We analyze an algorithm which iterates between
estimating local linear models of nonlinear system dynamics and performing
$\mathtt{iLQR}$-like policy updates. We demonstrate that this algorithm attains
sample complexity polynomial in relevant problem parameters, and, by
synthesizing locally stabilizing gains, overcomes exponential dependence in
problem horizon. Experimental results validate the performance of our
algorithm, and compare to natural deep-learning baselines.
( 2
min )
While Generative Adversarial Networks (GANs) have recently found applications
in image editing, most previous GAN-based image editing methods require
largescale datasets with semantic segmentation annotations for training, only
provide high level control, or merely interpolate between different images.
Previous researchers have proposed EditGAN for high-quality, high-precision
semantic image editing with limited semantic annotations by finding `editing
vectors'. However, it is noticed that there are many features that are not
highly associated with semantics, and EditGAN may fail on them. Based on the
orthogonality of latent space observed by EditGAN, we propose a method to
estimate editing vectors that do not rely on semantic segmentation nor
differentiable feature estimation network. Our method assumes that there is a
correlation between the intensity distribution of features and the distribution
of hidden vectors, and estimates the relationship between the above
distributions by sampling the feature intensity of the image corresponding to
several hidden vectors. We modified Linear Discriminant Analysis (LDA) to deal
with both binary feature editing and continuous feature editing. We then found
that this method has a good effect in processing features such as clothing type
and texture, skin color and hair.
( 2
min )
The article presents the torchosr package - a Python package compatible with
PyTorch library - offering tools and methods dedicated to Open Set Recognition
in Deep Neural Networks. The package offers two state-of-the-art methods in the
field, a set of functions for handling base sets and generation of derived sets
for the Open Set Recognition task (where some classes are considered unknown
and used only in the testing process) and additional tools to handle datasets
and methods. The main goal of the package proposal is to simplify and promote
the correct experimental evaluation, where experiments are carried out on a
large number of derivative sets with various Openness and class-to-category
assignments. The authors hope that state-of-the-art methods available in the
package will become a source of a correct and open-source implementation of the
relevant solutions in the domain.
( 2
min )
Dynamical mean-field theory is a powerful physics tool used to analyze the
typical behavior of neural networks, where neurons can be recurrently
connected, or multiple layers of neurons can be stacked. However, it is not
easy for beginners to access the essence of this tool and the underlying
physics. Here, we give a pedagogical introduction of this method in a
particular example of generic random neural networks, where neurons are
randomly and fully connected by correlated synapses and therefore the network
exhibits rich emergent collective dynamics. We also review related past and
recent important works applying this tool. In addition, a physically
transparent and alternative method, namely the dynamical cavity method, is also
introduced to derive exactly the same results. The numerical implementation of
solving the integro-differential mean-field equations is also detailed, with an
illustration of exploring the fluctuation dissipation theorem.
( 2
min )
Denoising diffusion models are a class of generative models which have
recently achieved state-of-the-art results across many domains. Gradual noise
is added to the data using a diffusion process, which transforms the data
distribution into a Gaussian. Samples from the generative model are then
obtained by simulating an approximation of the time reversal of this diffusion
initialized by Gaussian samples. Recent research has explored adapting
diffusion models for sampling and inference tasks. In this paper, we leverage
known connections to stochastic control akin to the F\"ollmer drift to extend
established neural network approximation results for the F\"ollmer drift to
denoising diffusion models and samplers.
( 2
min )
Localizing behaviors of neural networks to a subset of the network's
components or a subset of interactions between components is a natural first
step towards analyzing network mechanisms and possible failure modes. Existing
work is often qualitative and ad-hoc, and there is no consensus on the
appropriate way to evaluate localization claims. We introduce path patching, a
technique for expressing and quantitatively testing a natural class of
hypotheses expressing that behaviors are localized to a set of paths. We refine
an explanation of induction heads, characterize a behavior of GPT-2, and open
source a framework for efficiently running similar experiments.
( 2
min )
The concept of neuromuscular activity recognition using instantaneous
high-density surface electromyography (HD-sEMG) images opens up new avenues for
the development of more fluid and natural muscle-computer interfaces. However,
the existing approaches employed a very large deep convolutional neural network
(ConvNet) architecture and complex training schemes for HD-sEMG image
recognition, which requires the network architecture to be pre-trained on a
very large-scale labeled training dataset, as a result, it makes
computationally very expensive. To overcome this problem, we propose S-ConvNet
and All-ConvNet models, a simple yet efficient framework for learning
instantaneous HD-sEMG images from scratch for neuromuscular activity
recognition. Without using any pre-trained models, our proposed S-ConvNet and
All-ConvNet demonstrate very competitive recognition accuracy to the more
complex state of the art for neuromuscular activity recognition based on
instantaneous HD-sEMG images, while using a ~ 12 x smaller dataset and reducing
learning parameters to a large extent. The experimental results proved that the
S-ConvNet and All-ConvNet are highly effective for learning discriminative
features for instantaneous HD-sEMG image recognition especially in the data and
high-end resource constrained scenarios.
( 2
min )
In recent years, there has been a growing interest in the effects of data
poisoning attacks on data-driven control methods. Poisoning attacks are
well-known to the Machine Learning community, which, however, make use of
assumptions, such as cross-sample independence, that in general do not hold for
linear dynamical systems. Consequently, these systems require different attack
and detection methods than those developed for supervised learning problems in
the i.i.d.\ setting. Since most data-driven control algorithms make use of the
least-squares estimator, we study how poisoning impacts the least-squares
estimate through the lens of statistical testing, and question in what way data
poisoning attacks can be detected. We establish under which conditions the set
of models compatible with the data includes the true model of the system, and
we analyze different poisoning strategies for the attacker. On the basis of the
arguments hereby presented, we propose a stealthy data poisoning attack on the
least-squares estimator that can escape classical statistical tests, and
conclude by showing the efficiency of the proposed attack.
( 2
min )
Digital transformation in buildings accumulates massive operational data,
which calls for smart solutions to utilize these data to improve energy
performance. This study has proposed a solution, namely Deep Energy Twin, for
integrating deep learning and digital twins to better understand building
energy use and identify the potential for improving energy efficiency. Ontology
was adopted to create parametric digital twins to provide consistency of data
format across different systems in a building. Based on created digital twins
and collected data, deep learning methods were used for performing data
analytics to identify patterns and provide insights for energy optimization. As
a demonstration, a case study was conducted in a public historic building in
Norrk\"oping, Sweden, to compare the performance of state-of-the-art deep
learning architectures in building energy forecasting.
( 2
min )
In manufacturing settings, data collection and analysis are often a
time-consuming, challenging, and costly process. It also hinders the use of
advanced machine learning and data-driven methods which require a substantial
amount of offline training data to generate good results. It is particularly
challenging for small manufacturers who do not share the resources of a large
enterprise. Recently, with the introduction of the Internet of Things (IoT),
data can be collected in an integrated manner across the factory in real-time,
sent to the cloud for advanced analysis, and used to update the machine
learning model sequentially. Nevertheless, small manufacturers face two
obstacles in reaping the benefits of IoT: they may be unable to afford or
generate enough data to operate a private cloud, and they may be hesitant to
share their raw data with a public cloud. Federated learning (FL) is an
emerging concept of collaborative learning that can help small-scale industries
address these issues and learn from each other without sacrificing their
privacy. It can bring together diverse and geographically dispersed
manufacturers under the same analytics umbrella to create a win-win situation.
However, the widespread adoption of FL across multiple manufacturing
organizations remains a significant challenge. This study aims to review the
challenges and future directions of applying federated learning in the
manufacturing industry, with a specific emphasis on the perspectives of
Industry 4.0 and 5.0.
( 3
min )
In this work we introduce a self-supervised pretraining framework for
transformers on functional Magnetic Resonance Imaging (fMRI) data. First, we
pretrain our architecture on two self-supervised tasks simultaneously to teach
the model a general understanding of the temporal and spatial dynamics of human
auditory cortex during music listening. Our pretraining results are the first
to suggest a synergistic effect of multitask training on fMRI data. Second, we
finetune the pretrained models and train additional fresh models on a
supervised fMRI classification task. We observe significantly improved accuracy
on held-out runs with the finetuned models, which demonstrates the ability of
our pretraining tasks to facilitate transfer learning. This work contributes to
the growing body of literature on transformer architectures for pretraining and
transfer learning with fMRI data, and serves as a proof of concept for our
pretraining tasks and multitask pretraining on fMRI data.
( 2
min )
"Data is the new oil", in short, data would be the essential source of the
ongoing fourth industrial revolution, which has led some commentators to
assimilate too quickly the quantity of data to a source of wealth in itself,
and consider the development of big data as an quasi direct cause of profit.
Human resources management is not escaping this trend, and the accumulation of
large amounts of data on employees is perceived by some entrepreneurs as a
necessary and sufficient condition for the construction of predictive models of
complex work behaviors such as absenteeism or job performance. In fact, the
analogy is somewhat misleading: unlike oil, there are no major issues here
concerning the production of data (whose flows are generated continuously and
at low cost by various information …
( 3
min )
The detection of malicious websites has become a critical issue in
cybersecurity. Therefore, this paper offers a comprehensive review of
data-driven methods for detecting malicious websites. Traditional approaches
and their limitations are discussed, followed by an overview of data-driven
approaches. The paper establishes the data-feature-model-extension pipeline and
the latest research developments of data-driven approaches, including data
preprocessing, feature extraction, model construction and technology extension.
Specifically, this paper compares methods using deep learning models proposed
in recent years. Furthermore, the paper follows the
data-feature-model-extension pipeline to discuss the challenges together with
some future directions of data-driven methods in malicious website detection.
( 2
min )
Backpropagation (BP) is the most important gradient estimation method for
training neural networks in deep learning. However, the literature shows that
neural networks trained by BP are vulnerable to adversarial attacks. We develop
the likelihood ratio (LR) method, a new gradient estimation method, for
training a broad range of neural network architectures, including convolutional
neural networks, recurrent neural networks, graph neural networks, and spiking
neural networks, without recursive gradient computation. We propose three
methods to efficiently reduce the variance of the gradient estimation in the
neural network training process. Our experiments yield numerical results for
training different neural networks on several datasets. All results demonstrate
that the LR method is effective for training various neural networks and
significantly improves the robustness of the neural networks under adversarial
attacks relative to the BP method.
( 2
min )
Neurosymbolic AI deals with models that combine symbolic processing, like
classic AI, and neural networks, as it's a very established area. These models
are emerging as an effort toward Artificial General Intelligence (AGI) by both
exploring an alternative to just increasing datasets' and models' sizes and
combining Learning over the data distribution, Reasoning on prior and learned
knowledge, and by symbiotically using them. This survey investigates research
papers in this area during recent years and brings classification and
comparison between the presented models as well as applications.
( 2
min )
In this work, we propose a deep learning (DL)-based constitutive model for
investigating the cyclic viscoelastic-viscoplastic-damage behavior of
nanoparticle/epoxy nanocomposites with moisture content. For this, a long
short-term memory network is trained using a combined framework of a sampling
technique and a perturbation method. The training framework, along with the
training data generated by an experimentally validated
viscoelastic-viscoplastic model, enables the DL model to accurately capture the
rate-dependent stress-strain relationship and consistent tangent moduli. In
addition, the DL-based constitutive model is implemented into finite element
analysis. Finite element simulations are performed to study the effect of load
rate and moisture content on the force-displacement response of nanoparticle/
epoxy samples. Numerical examples show that the computational efficiency of the
DL model depends on the loading condition and is significantly higher than the
conventional constitutive model. Furthermore, comparing numerical results and
experimental data demonstrates good agreement with different nanoparticle and
moisture contents.
( 2
min )
PiML (read $\pi$-ML, /`pai.`em.`el/) is an integrated and open-access Python
toolbox for interpretable machine learning model development and model
diagnostics. It is designed with machine learning workflows in both low-code
and high-code modes, including data pipeline, model training, model
interpretation and explanation, and model diagnostics and comparison. The
toolbox supports a growing list of interpretable models (e.g. GAM, GAMI-Net,
XGB2) with inherent local and/or global interpretability. It also supports
model-agnostic explainability tools (e.g. PFI, PDP, LIME, SHAP) and a powerful
suite of model-agnostic diagnostics (e.g. weakness, uncertainty, robustness,
fairness). Integration of PiML models and tests to existing MLOps platforms for
quality assurance are enabled by flexible high-code APIs. Furthermore, PiML
toolbox comes with a comprehensive user guide and hands-on examples, including
the applications for model development and validation in banking. The project
is available at https://github.com/SelfExplainML/PiML-Toolbox.
( 2
min )
Recent years have witnessed a growth in mathematics for deep learning--which
seeks a deeper understanding of the concepts of deep learning with mathematics
and explores how to make it more robust--and deep learning for mathematics,
where deep learning algorithms are used to solve problems in mathematics. The
latter has popularised the field of scientific machine learning where deep
learning is applied to problems in scientific computing. Specifically, more and
more neural network architectures have been developed to solve specific classes
of partial differential equations (PDEs). Such methods exploit properties that
are inherent to PDEs and thus solve the PDEs better than standard feed-forward
neural networks, recurrent neural networks, or convolutional neural networks.
This has had a great impact in the area of mathematical modeling where
parametric PDEs are widely used to model most natural and physical processes
arising in science and engineering. In this work, we review such methods as
well as their extensions for parametric studies and for solving the related
inverse problems. We equally proceed to show their relevance in some industrial
applications.
( 2
min )
We characterise the learning of a mixture of two clouds of data points with
generic centroids via empirical risk minimisation in the high dimensional
regime, under the assumptions of generic convex loss and convex regularisation.
Each cloud of data points is obtained by sampling from a possibly uncountable
superposition of Gaussian distributions, whose variance has a generic
probability density $\varrho$. Our analysis covers therefore a large family of
data distributions, including the case of power-law-tailed distributions with
no covariance. We study the generalisation performance of the obtained
estimator, we analyse the role of regularisation, and the dependence of the
separability transition on the distribution scale parameters.
( 2
min )
This work concerns controlling the false discovery rate (FDR) in networks
under communication constraints. We present sample-and-forward, a flexible and
communication-efficient version of the Benjamini-Hochberg (BH) procedure for
multihop networks with general topologies. Our method evidences that the nodes
in a network do not need to communicate p-values to each other to achieve a
decent statistical power under the global FDR control constraint. Consider a
network with a total of $m$ p-values, our method consists of first sampling the
(empirical) CDF of the p-values at each node and then forwarding
$\mathcal{O}(\log m)$ bits to its neighbors. Under the same assumptions as for
the original BH procedure, our method has both the provable finite-sample FDR
control as well as competitive empirical detection power, even with a few
samples at each node. We provide an asymptotic analysis of power under a
mixture model assumption on the p-values.
( 2
min )
We characterise the learning of a mixture of two clouds of data points with
generic centroids via empirical risk minimisation in the high dimensional
regime, under the assumptions of generic convex loss and convex regularisation.
Each cloud of data points is obtained by sampling from a possibly uncountable
superposition of Gaussian distributions, whose variance has a generic
probability density $\varrho$. Our analysis covers therefore a large family of
data distributions, including the case of power-law-tailed distributions with
no covariance. We study the generalisation performance of the obtained
estimator, we analyse the role of regularisation, and the dependence of the
separability transition on the distribution scale parameters.
( 2
min )
D-Adaptation is an approach to automatically setting the learning rate which
asymptotically achieves the optimal rate of convergence for minimizing convex
Lipschitz functions, with no back-tracking or line searches, and no additional
function value or gradient evaluations per step. Our approach is the first
hyper-parameter free method for this class without additional multiplicative
log factors in the convergence rate. We present extensive experiments for SGD
and Adam variants of our method, where the method automatically matches
hand-tuned learning rates across more than a dozen diverse machine learning
problems, including large-scale vision and language problems.
An open-source implementation is available.
( 2
min )
This study explores the number of neurons required for a Rectified Linear
Unit (ReLU) neural network to approximate multivariate monomials. We establish
an exponential lower bound on the complexity of any shallow network
approximating the product function over a general compact domain. We also
demonstrate this lower bound doesn't apply to normalized Lipschitz monomials
over the unit cube. These findings suggest that shallow ReLU networks
experience the curse of dimensionality when expressing functions with a
Lipschitz parameter scaling with the dimension of the input, and that the
expressive power of neural networks is more dependent on their depth rather
than overall complexity.
( 2
min )
We present a novel framework for distributionally robust optimization (DRO),
called cost-aware DRO (CADRO). The key idea of CADRO is to exploit the cost
structure in the design of the ambiguity set to reduce conservatism.
Particularly, the set specifically constrains the worst-case distribution along
the direction in which the expected cost of an approximate solution increases
most rapidly. We prove that CADRO provides both a high-confidence upper bound
and a consistent estimator of the out-of-sample expected cost, and show
empirically that it produces solutions that are substantially less conservative
than existing DRO methods, while providing the same guarantees.
( 2
min )
Denoising diffusion models are a class of generative models which have
recently achieved state-of-the-art results across many domains. Gradual noise
is added to the data using a diffusion process, which transforms the data
distribution into a Gaussian. Samples from the generative model are then
obtained by simulating an approximation of the time reversal of this diffusion
initialized by Gaussian samples. Recent research has explored adapting
diffusion models for sampling and inference tasks. In this paper, we leverage
known connections to stochastic control akin to the F\"ollmer drift to extend
established neural network approximation results for the F\"ollmer drift to
denoising diffusion models and samplers.
( 2
min )
A common pipeline in learning-based control is to iteratively estimate a
model of system dynamics, and apply a trajectory optimization algorithm -
e.g.~$\mathtt{iLQR}$ - on the learned model to minimize a target cost. This
paper conducts a rigorous analysis of a simplified variant of this strategy for
general nonlinear systems. We analyze an algorithm which iterates between
estimating local linear models of nonlinear system dynamics and performing
$\mathtt{iLQR}$-like policy updates. We demonstrate that this algorithm attains
sample complexity polynomial in relevant problem parameters, and, by
synthesizing locally stabilizing gains, overcomes exponential dependence in
problem horizon. Experimental results validate the performance of our
algorithm, and compare to natural deep-learning baselines.
( 2
min )
While Convolutional Neural Networks (CNNs) have long been investigated and
applied, as well as theorized, we aim to provide a slightly different
perspective into their nature -- through the perspective of their Hessian maps.
The reason is that the loss Hessian captures the pairwise interaction of
parameters and therefore forms a natural ground to probe how the architectural
aspects of CNN get manifested in its structure and properties. We develop a
framework relying on Toeplitz representation of CNNs, and then utilize it to
reveal the Hessian structure and, in particular, its rank. We prove tight upper
bounds (with linear activations), which closely follow the empirical trend of
the Hessian rank and hold in practice in more general settings. Overall, our
work generalizes and establishes the key insight that, even in CNNs, the
Hessian rank grows as the square root of the number of parameters.
( 2
min )
ChatGPT is a sophisticated language model that has taken the world by storm. With its advanced natural language processing capabilities and…
( 11
min )
Some reactions to the latest AI news and developments, along with some AI-generated artwork. Follow the channel, to get updates on posts.
( 13
min )
Leo Anthony Celi invites industry to broaden its focus in gathering and analyzing clinical data for every population.
( 9
min )
Announcements LLM success depends on quality, transparent data Everyone from writers to coders wonder if their job is in jeopardy as prognosticators say generative AI tools will take over business in the coming years. Of course, these large language model chatbots are still unreliable, and certainly can’t be trusted to complete jobs as well as… Read More »DSC Weekly 16 May 2023 – LLM success depends on quality, transparent data
The post DSC Weekly 16 May 2023 – LLM success depends on quality, transparent data appeared first on Data Science Central.
( 19
min )
Image sourced from striim.com There’s one universal truth for every modern organization. It doesn’t matter whether you’re starting a business or already established: to succeed, you need data. Of course, not just any data will do. For strong data-driven decision-making, you also need the best insights. Thankfully, due to data analytics tools, businesses of all… Read More »6 Reasons Real-Time Data Analytics is Beneficial for Your Business
The post 6 Reasons Real-Time Data Analytics is Beneficial for Your Business appeared first on Data Science Central.
( 24
min )
In today’s technological world, data is everything. It can inform our marketing decisions, improve product creation, boost internal processes, and more. For an online business, having the best possible data is key to success. But simply having data isn’t enough. To obtain useful information, you need to understand your data. That’s where web data analytics… Read More »5 Ways to Use Analytics to Inform Website Development Decisions
The post 5 Ways to Use Analytics to Inform Website Development Decisions appeared first on Data Science Central.
( 24
min )
During the past decade, the publishing industry has undergone significant transformations due to the development of digital platforms and the widespread availability of user-generated content. Although these advancements have enabled a greater availability of information and a more diverse perspective, they have also presented challenges when it comes to ensuring that the content adheres to… Read More »Publishing Industry: The Extreme Crucial Role of AI in Content Moderation
The post Publishing Industry: The Extreme Crucial Role of AI in Content Moderation appeared first on Data Science Central.
( 22
min )
Today we are excited to announce that Together Computer’s GPT-NeoXT-Chat-Base-20B language foundation model is available for customers using Amazon SageMaker JumpStart. GPT-NeoXT-Chat-Base-20B is an open-source model to build conversational bots. You can easily try out this model and use it with JumpStart. JumpStart is the machine learning (ML) hub of Amazon SageMaker that provides access […]
( 12
min )
This research was accepted by the IEEE/ACM International Conference on Software Engineering (ICSE), which is a forum for researchers, practitioners, and educators to gather, present, and discuss the most recent innovations, trends, experiences, and issues in the field of software engineering. The Microsoft 365 Systems Innovation research group has a paper accepted at the 45th […]
The post Large-language models for automatic cloud incident management appeared first on Microsoft Research.
( 11
min )
Ten thousand years after the last woolly mammoths vanished with the last Ice Age, a team of computational biologists is on a mission to bring them back within five years. Led by synthetic biology pioneer George Church, Colossal Biosciences is also seeking to return the dodo bird and Tasmanian tiger, as well as help save Read article >
( 7
min )
Chip manufacturing is an “ideal application” for NVIDIA accelerated and AI computing, NVIDIA founder and CEO Jensen Huang said Tuesday. Detailing how the latest advancements in computing are accelerating “the world’s most important industry,” Huang spoke at ITF World 2023 semiconductor conference in Antwerp, Belgium. Huang delivered his remarks via video to a gathering of Read article >
( 7
min )
Data Science is a popular as well as vast field; till date, there are a lot of opportunities in this field, and most people, whether they…
( 25
min )
Introduction
( 17
min )
This is a guest post co-authored by Nafi Ahmet Turgut, Mutlu Polatcan, Pınar Baki, Mehmet İkbal Özmen, Hasan Burak Yel, and Hamza Akyıldız from Getir. Getir is the pioneer of ultrafast grocery delivery. The tech company has revolutionized last-mile delivery with its “groceries in minutes” delivery proposition. Getir was founded in 2015 and operates in […]
( 8
min )
Amazon Textract is a machine learning (ML) service that automatically extracts text, handwriting, and data from any document or image. To make it simpler to evaluate the capabilities of Amazon Textract, we have launched a new Bulk Document Uploader feature on the Amazon Textract console that enables you to quickly process your own set of […]
( 7
min )
$\partial\mathbb{B}$ nets are differentiable neural networks that learn
discrete boolean-valued functions by gradient descent. $\partial\mathbb{B}$
nets have two semantically equivalent aspects: a differentiable soft-net, with
real weights, and a non-differentiable hard-net, with boolean weights. We train
the soft-net by backpropagation and then `harden' the learned weights to yield
boolean weights that bind with the hard-net. The result is a learned discrete
function. `Hardening' involves no loss of accuracy, unlike existing approaches
to neural network binarization. Preliminary experiments demonstrate that
$\partial\mathbb{B}$ nets achieve comparable performance on standard machine
learning problems yet are compact (due to 1-bit weights) and interpretable (due
to the logical nature of the learnt functions).
( 2
min )
Understanding material surfaces and interfaces is vital in applications like
catalysis or electronics. Ab initio simulations, combining energies from
electronic structure with statistical mechanics, can, in principle, predict the
structure of material surfaces as a function of thermodynamic variables.
However, accurate energy simulations are prohibitive when coupled to the vast
phase space that must be statistically sampled. Here, we present a bi-faceted
computational loop to predict surface phase diagrams of multi-component
materials that accelerates both the energy scoring and statistical sampling
methods. Fast, scalable, and data-efficient machine learning interatomic
potentials are trained on high-throughput density-functional theory
calculations through closed-loop active learning. Markov-chain Monte Carlo
sampling in the semi-grand canonical ensemble is enabled by using virtual
surface sites. The predicted surfaces for GaN(0001) and SrTiO3(001) are in
agreement with past work and suggest that the proposed strategy can model
complex material surfaces and discover previously unreported surface
terminations.
( 2
min )
Pre-trained language models (PLMs) are known to improve the generalization
performance of natural language understanding models by leveraging large
amounts of data during the pre-training phase. However, the out-of-distribution
(OOD) generalization problem remains a challenge in many NLP tasks, limiting
the real-world deployment of these methods. This paper presents the first
attempt at creating a unified benchmark named \method for evaluating OOD
robustness in NLP models, highlighting the importance of OOD robustness and
providing insights on how to measure the robustness of a model and how to
improve it. The benchmark includes 13 publicly available datasets for OOD
testing, and evaluations are conducted on 8 classic NLP tasks over 21 popularly
used PLMs, including GPT-3 and GPT-3.5. Our findings confirm the need for
improved OOD accuracy in NLP tasks, as significant performance degradation was
observed in all settings compared to in-distribution (ID) accuracy.
( 2
min )
Given data ${\rm X}\in\mathbb{R}^{n\times d}$ and labels
$\mathbf{y}\in\mathbb{R}^{n}$ the goal is find $\mathbf{w}\in\mathbb{R}^d$ to
minimize $\Vert{\rm X}\mathbf{w}-\mathbf{y}\Vert^2$. We give a polynomial
algorithm that, \emph{oblivious to $\mathbf{y}$}, throws out $n/(d+\sqrt{n})$
data points and is a $(1+d/n)$-approximation to optimal in expectation. The
motivation is tight approximation with reduced label complexity (number of
labels revealed). We reduce label complexity by $\Omega(\sqrt{n})$. Open
question: Can label complexity be reduced by $\Omega(n)$ with tight
$(1+d/n)$-approximation?
( 2
min )
Breast cancer early detection is crucial for improving patient outcomes. The
Institut Catal\`a de la Salut (ICS) has launched the DigiPatICS project to
develop and implement artificial intelligence algorithms to assist with the
diagnosis of cancer. In this paper, we propose a new approach for facing the
color normalization problem in HER2-stained histopathological images of breast
cancer tissue, posed as an style transfer problem. We combine the Color
Deconvolution technique with the Pix2Pix GAN network to present a novel
approach to correct the color variations between different HER2 stain brands.
Our approach focuses on maintaining the HER2 score of the cells in the
transformed images, which is crucial for the HER2 analysis. Results demonstrate
that our final model outperforms the state-of-the-art image style transfer
methods in maintaining the cell classes in the transformed images and is as
effective as them in generating realistic images.
( 2
min )
Over the last few years, the number of works about deep learning applied to
the medical field has increased enormously. The necessity of a rigorous
assessment of these models is required to explain these results to all people
involved in medical exams. A recent field in the machine learning area is
explainable artificial intelligence, also known as XAI, which targets to
explain the results of such black box models to permit the desired assessment.
This survey analyses several recent studies in the XAI field applied to medical
diagnosis research, allowing some explainability of the machine learning
results in several different diseases, such as cancers and COVID-19.
( 2
min )
This paper proposes an online, provably robust, and scalable Bayesian
approach for changepoint detection. The resulting algorithm has key advantages
over previous work: it provides provable robustness by leveraging the
generalised Bayesian perspective, and also addresses the scalability issues of
previous attempts. Specifically, the proposed generalised Bayesian formalism
leads to conjugate posteriors whose parameters are available in closed form by
leveraging diffusion score matching. The resulting algorithm is exact, can be
updated through simple algebra, and is more than 10 times faster than its
closest competitor.
( 2
min )
In the misspecified kernel ridge regression problem, researchers usually
assume the underground true function $f_{\rho}^{*} \in [\mathcal{H}]^{s}$, a
less-smooth interpolation space of a reproducing kernel Hilbert space (RKHS)
$\mathcal{H}$ for some $s\in (0,1)$. The existing minimax optimal results
require $\|f_{\rho}^{*}\|_{L^{\infty}}
\alpha_{0}$ where $\alpha_{0}\in (0,1)$ is the embedding index, a constant
depending on $\mathcal{H}$. Whether the KRR is optimal for all $s\in (0,1)$ is
an outstanding problem lasting for years. In this paper, we show that KRR is
minimax optimal for any $s\in (0,1)$ when the $\mathcal{H}$ is a Sobolev RKHS.
( 2
min )
Anonymization techniques based on obfuscating the quasi-identifiers by means
of value generalization hierarchies are widely used to achieve preset levels of
privacy. To prevent different types of attacks against database privacy it is
necessary to apply several anonymization techniques beyond the classical
k-anonymity or $\ell$-diversity. However, the application of these methods is
directly connected to a reduction of their utility in prediction and decision
making tasks. In this work we study four classical machine learning methods
currently used for classification purposes in order to analyze the results as a
function of the anonymization techniques applied and the parameters selected
for each of them. The performance of these models is studied when varying the
value of k for k-anonymity and additional tools such as $\ell$-diversity,
t-closeness and $\delta$-disclosure privacy are also deployed on the well-known
adult dataset.
( 2
min )
The Schr\"odinger bridge problem (SBP) is gaining increasing attention in
generative modeling and showing promising potential even in comparison with the
score-based generative models (SGMs). SBP can be interpreted as an
entropy-regularized optimal transport problem, which conducts projections onto
every other marginal alternatingly. However, in practice, only approximated
projections are accessible and their convergence is not well understood. To
fill this gap, we present a first convergence analysis of the Schr\"odinger
bridge algorithm based on approximated projections. As for its practical
applications, we apply SBP to probabilistic time series imputation by
generating missing values conditioned on observed data. We show that optimizing
the transport cost improves the performance and the proposed algorithm achieves
the state-of-the-art result in healthcare and environmental data while
exhibiting the advantage of exploring both temporal and feature patterns in
probabilistic time series imputation.
( 2
min )
The use of deep learning approaches for image reconstruction is of
contemporary interest in radiology, especially for approaches that solve
inverse problems associated with imaging. In deployment, these models may be
exposed to input distributions that are widely shifted from training data, due
in part to data biases or drifts. We propose a metric based on local Lipschitz
determined from a single trained model that can be used to estimate the model
uncertainty for image reconstructions. We demonstrate a monotonic relationship
between the local Lipschitz value and Mean Absolute Error and show that this
method can be used to provide a threshold that determines whether a given DL
reconstruction approach was well suited to the task. Our uncertainty estimation
method can be used to identify out-of-distribution test samples, relate
information regarding epistemic uncertainties, and guide proper data
augmentation. Quantifying uncertainty of learned reconstruction approaches is
especially pertinent to the medical domain where reconstructed images must
remain diagnostically accurate.
( 2
min )
Recent works successfully leveraged Large Language Models' (LLM) abilities to
capture abstract knowledge about world's physics to solve decision-making
problems. Yet, the alignment between LLMs' knowledge and the environment can be
wrong and limit functional competence due to lack of grounding. In this paper,
we study an approach (named GLAM) to achieve this alignment through functional
grounding: we consider an agent using an LLM as a policy that is progressively
updated as the agent interacts with the environment, leveraging online
Reinforcement Learning to improve its performance to solve goals. Using an
interactive textual environment designed to study higher-level forms of
functional grounding, and a set of spatial and navigation tasks, we study
several scientific questions: 1) Can LLMs boost sample efficiency for online
learning of various RL tasks? 2) How can it boost different forms of
generalization? 3) What is the impact of online learning? We study these
questions by functionally grounding several variants (size, architecture) of
FLAN-T5.
( 2
min )
This paper discusses the feasibility of continuously training the CLIP model
through streaming data. Then, by tracking the directional changes of the
representation vectors in the continuously updated CLIP model, we explore and
summarize these spatial variations as Spatial Disorder (SD), which can be
divided into Intra-modal Rotation and Inter-modal Deviation. Moreover, we
demonstrate how intra-modal rotation and inter-modal deviation lead to a
performance decline for CLIP on cross-modal retrieval tasks in both empirically
and theoretically. To alleviate the spatial disorder, we propose a simple yet
effective continual learning framework Mod-X: Maintain off-diagonal
information-matriX. The experiments (in Section \ref{method}, \ref{experiments}
and Appendix \ref{Appendix_to_experiments}) on commonly used datasets with
different scales and scopes have illustrated the effectiveness of our method.
( 2
min )
Critical decisions like loan approvals, medical interventions, and college
admissions are guided by predictions made in the presence of uncertainty. In
this paper, we prove that uncertainty has a disparate impact. While it imparts
errors across all demographic groups, the types of errors vary systematically:
Groups with higher average outcomes are typically assigned higher false
positive rates, while those with lower average outcomes are assigned higher
false negative rates. We show that additional data acquisition can eliminate
the disparity and broaden access to opportunity. The strategy, which we call
Affirmative Information, could stand as an alternative to Affirmative Action.
( 2
min )
In this work, we focus on the communication aspect of decentralized learning,
which involves multiple agents training a shared machine learning model using
decentralized stochastic gradient descent (D-SGD) over distributed data. In
particular, we investigate the impact of broadcast transmission and
probabilistic random access policy on the convergence performance of D-SGD,
considering the broadcast nature of wireless channels and the link dynamics in
the communication topology. Our results demonstrate that optimizing the access
probability to maximize the expected number of successful links is a highly
effective strategy for accelerating the system convergence.
( 2
min )
Existing question answering methods often assume that the input content
(e.g., documents or videos) is always accessible to solve the task.
Alternatively, memory networks were introduced to mimic the human process of
incremental comprehension and compression of the information in a
fixed-capacity memory. However, these models only learn how to maintain memory
by backpropagating errors in the answers through the entire network. Instead,
it has been suggested that humans have effective mechanisms to boost their
memorization capacities, such as rehearsal and anticipation. Drawing
inspiration from these, we propose a memory model that performs rehearsal and
anticipation while processing inputs to memorize important information for
solving question answering tasks from streaming data. The proposed mechanisms
are applied self-supervised during training through masked modeling tasks
focused on coreference information. We validate our model on a short-sequence
(bAbI) dataset as well as large-sequence textual (NarrativeQA) and video
(ActivityNet-QA) question answering datasets, where it achieves substantial
improvements over previous memory network approaches. Furthermore, our ablation
study confirms the proposed mechanisms' importance for memory models.
( 2
min )
Diffusion models were initially developed for text-to-image generation and
are now being utilized to generate high quality synthetic images. Preceded by
GANs, diffusion models have shown impressive results using various evaluation
metrics. However, commonly used metrics such as FID and IS are not suitable for
determining whether diffusion models are simply reproducing the training
images. Here we train StyleGAN and diffusion models, using BRATS20 and BRATS21
datasets, to synthesize brain tumor images, and measure the correlation between
the synthetic images and all training images. Our results show that diffusion
models are much more likely to memorize the training images, especially for
small datasets. Researchers should be careful when using diffusion models for
medical imaging, if the final goal is to share the synthetic images.
( 2
min )
We present GPS++, a hybrid Message Passing Neural Network / Graph Transformer
model for molecular property prediction. Our model integrates a well-tuned
local message passing component and biased global attention with other key
ideas from prior literature to achieve state-of-the-art results on large-scale
molecular dataset PCQM4Mv2. Through a thorough ablation study we highlight the
impact of individual components and find that nearly all of the model's
performance can be maintained without any use of global self-attention, showing
that message passing is still a competitive approach for 3D molecular property
prediction despite the recent dominance of graph transformers. We also find
that our approach is significantly more accurate than prior art when 3D
positional information is not available.
( 2
min )
This field case study aims to address the challenge of accurately predicting
petrophysical properties in heterogeneous reservoir formations, which can
significantly impact reservoir performance predictions. The study employed
three machine learning algorithms, namely Artificial Neural Network (ANN),
Random Forest Classifier (RFC), and Support Vector Machine (SVM), to predict
permeability log from conventional logs and match it with core data. The
primary objective of this study was to compare the effectiveness of the three
machine learning algorithms in predicting permeability and determine the
optimal prediction method. The study utilized the Flow Zone Indicator (FZI)
rock typing technique to understand the factors influencing reservoir quality.
The findings will be used to improve reservoir simulation and locate future
wells more accurately. The study concluded that the FZI approach and machine
learning algorithms are effective in predicting permeability log and improving
reservoir performance predictions.
( 2
min )
Classical reinforcement learning (RL) aims to optimize the expected
cumulative reward. In this work, we consider the RL setting where the goal is
to optimize the quantile of the cumulative reward. We parameterize the policy
controlling actions by neural networks, and propose a novel policy gradient
algorithm called Quantile-Based Policy Optimization (QPO) and its variant
Quantile-Based Proximal Policy Optimization (QPPO) for solving deep RL problems
with quantile objectives. QPO uses two coupled iterations running at different
timescales for simultaneously updating quantiles and policy parameters, whereas
QPPO is an off-policy version of QPO that allows multiple updates of parameters
during one simulation episode, leading to improved algorithm efficiency. Our
numerical results indicate that the proposed algorithms outperform the existing
baseline algorithms under the quantile criterion.
( 2
min )
With the publication of DINO, a variant of the Detection Transformer (DETR),
Detection Transformers are breaking the record in the object detection
benchmark with the merits of their end-to-end design and scalability. However,
the extension of DETR to oriented object detection has not been thoroughly
studied although more benefits from its end-to-end architecture are expected
such as removing NMS and anchor-related costs. In this paper, we propose a
first strong DINO-based baseline for oriented object detection. We found that
straightforward employment of DETRs for oriented object detection does not
guarantee non-duplicate prediction, and propose a simple cost to mitigate this.
Furthermore, we introduce a novel denoising strategy that uses Hungarian
matching to filter redundant noised queries and query alignment to preserve
matching consistency between Transformer decoder layers. Our proposed model
outperforms previous rotated DETRs and other counterparts, achieving
state-of-the-art performance in DOTA-v1.0/v1.5/v2.0, and DIOR-R benchmarks.
( 2
min )
We propose the GFlowNets with Human Feedback (GFlowHF) framework to improve
the exploration ability when training AI models. For tasks where the reward is
unknown, we fit the reward function through human evaluations on different
trajectories. The goal of GFlowHF is to learn a policy that is strictly
proportional to human ratings, instead of only focusing on human favorite
ratings like RLHF. Experiments show that GFlowHF can achieve better exploration
ability than RLHF.
( 2
min )
We present the Hierarchical Mixture Networks (HINT), a model family for
efficient and accurate coherent forecasting. We specialize the networks on the
task via a multivariate mixture optimized with composite likelihood and made
coherent via bootstrap reconciliation. Additionally, we robustify the networks
to stark time series scale variations, incorporating normalized feature
extraction and recomposition of output scales within their architecture. We
demonstrate 8% sCRPS improved accuracy across five datasets compared to the
existing state-of-the-art. We conduct ablation studies on our model's
components and extensively investigate the theoretical properties of the
multivariate mixture. HINT's code is available at this
https://github.com/Nixtla/neuralforecast.
( 2
min )
This work proposes the use of 3D convolutional variational autoencoders
(CVAEs) to trace the changes and symptomatology produced by neurodegeneration
in Parkinson's disease (PD). In this work, we present a novel approach to
detect and quantify changes in dopamine transporter (DaT) concentration and its
spatial patterns using 3D CVAEs on Ioflupane (FPCIT) imaging. Our approach
leverages the power of deep learning to learn a low-dimensional representation
of the brain imaging data, which then is linked to different symptom categories
using regression algorithms. We demonstrate the effectiveness of our approach
on a dataset of PD patients and healthy controls, and show that general
symptomatology (UPDRS) is linked to a d-dimensional decomposition via the CVAE
with R2>0.25. Our work shows the potential of representation learning not only
in early diagnosis but in understanding neurodegeneration processes and
symptomatology.
( 2
min )
This paper proposes an online, provably robust, and scalable Bayesian
approach for changepoint detection. The resulting algorithm has key advantages
over previous work: it provides provable robustness by leveraging the
generalised Bayesian perspective, and also addresses the scalability issues of
previous attempts. Specifically, the proposed generalised Bayesian formalism
leads to conjugate posteriors whose parameters are available in closed form by
leveraging diffusion score matching. The resulting algorithm is exact, can be
updated through simple algebra, and is more than 10 times faster than its
closest competitor.
( 2
min )
We present the Hierarchical Mixture Networks (HINT), a model family for
efficient and accurate coherent forecasting. We specialize the networks on the
task via a multivariate mixture optimized with composite likelihood and made
coherent via bootstrap reconciliation. Additionally, we robustify the networks
to stark time series scale variations, incorporating normalized feature
extraction and recomposition of output scales within their architecture. We
demonstrate 8% sCRPS improved accuracy across five datasets compared to the
existing state-of-the-art. We conduct ablation studies on our model's
components and extensively investigate the theoretical properties of the
multivariate mixture. HINT's code is available at this
https://github.com/Nixtla/neuralforecast.
( 2
min )
Critical decisions like loan approvals, medical interventions, and college
admissions are guided by predictions made in the presence of uncertainty. In
this paper, we prove that uncertainty has a disparate impact. While it imparts
errors across all demographic groups, the types of errors vary systematically:
Groups with higher average outcomes are typically assigned higher false
positive rates, while those with lower average outcomes are assigned higher
false negative rates. We show that additional data acquisition can eliminate
the disparity and broaden access to opportunity. The strategy, which we call
Affirmative Information, could stand as an alternative to Affirmative Action.
( 2
min )
In Part 1 of the series “AI for Everyone: Learn How to Think Like a Data Scientist”, we discussed that for AI to reach its full economic and societal potential, we must educate and empower everyone to actively participate in the design, application, and management of meaningful, relevant, and responsible AI. We discussed the role… Read More »AI for Everyone: Learn How to Think Like a Data Scientist – Part 2
The post AI for Everyone: Learn How to Think Like a Data Scientist – Part 2 appeared first on Data Science Central.
( 19
min )
Deep machine learning models including Convolutional Neural Networks (CNN)
have been successful in the detection of Mild Cognitive Impairment (MCI) using
medical images, questionnaires, and videos. This paper proposes a novel
Multi-branch Classifier-Video Vision Transformer (MC-ViViT) model to
distinguish MCI from those with normal cognition by analyzing facial features.
The data comes from the I-CONECT, a behavioral intervention trial aimed at
improving cognitive function by providing frequent video chats. MC-ViViT
extracts spatiotemporal features of videos in one branch and augments
representations by the MC module. The I-CONECT dataset is challenging as the
dataset is imbalanced containing Hard-Easy and Positive-Negative samples, which
impedes the performance of MC-ViViT. We propose a loss function for Hard-Easy
and Positive-Negative Samples (HP Loss) by combining Focal loss and AD-CORRE
loss to address the imbalanced problem. Our experimental results on the
I-CONECT dataset show the great potential of MC-ViViT in predicting MCI with a
high accuracy of 90.63\% accuracy on some of the interview videos.
( 2
min )
The use of Artificial Intelligence (AI) has become increasingly prevalent in the modern world, seeing its potential to drastically improve…
( 25
min )
Amazon SageMaker comes with two options to spin up fully managed notebooks for exploring data and building machine learning (ML) models. The first option is fast start, collaborative notebooks accessible within Amazon SageMaker Studio—a fully integrated development environment (IDE) for machine learning. You can quickly launch notebooks in Studio, easily dial up or down the […]
( 9
min )
Amazon Kendra is an intelligent search service powered by machine learning (ML). Amazon Kendra reimagines enterprise search for your websites and applications so your employees and customers can easily find the content they’re looking for, even when it’s scattered across multiple locations and content repositories within your organization. Keywords or natural language questions can be […]
( 7
min )
EuroSys 2023 is the premier systems conference in Europe, and 2023 marks its 18th edition. Sponsored by ACM SIGOPS Europe and hosted May 8 to May 12, the conference covers a wide range of topics, including operating systems, real-time and networked systems, storage and middleware, and distributed, parallel, and embedded computing, as well as their […]
The post Microsoft at EuroSys 2023: Systems innovation across the stack to help support an easier, faster, safer, and smarter cloud appeared first on Microsoft Research.
( 11
min )
We consider the upper confidence bound strategy for Gaussian multi-armed
bandits with known control horizon sizes $N$ and build its limiting description
with a system of stochastic differential equations and ordinary differential
equations. Rewards for the arms are assumed to have unknown expected values and
known variances. A set of Monte-Carlo simulations was performed for the case of
close distributions of rewards, when mean rewards differ by the magnitude of
order $N^{-1/2}$, as it yields the highest normalized regret, to verify the
validity of the obtained description. The minimal size of the control horizon
when the normalized regret is not noticeably larger than maximum possible was
estimated.
( 2
min )
Due to the huge amount of parameters, fine-tuning of pretrained language
models (PLMs) is prone to overfitting in the low resource scenarios. In this
work, we present a novel method that operates on the hidden representations of
a PLM to reduce overfitting. During fine-tuning, our method inserts random
autoencoders between the hidden layers of a PLM, which transform activations
from the previous layers into multi-view compressed representations before
feeding them into the upper layers. The autoencoders are plugged out after
fine-tuning, so our method does not add extra parameters or increase
computation cost during inference. Our method demonstrates promising
performance improvement across a wide range of sequence- and token-level
low-resource NLP tasks.
( 2
min )
The Nash Equilibrium (NE) estimation in bidding games of electricity markets
is the key concern of both generation companies (GENCOs) for bidding strategy
optimization and the Independent System Operator (ISO) for market surveillance.
However, existing methods for NE estimation in emerging modern electricity
markets (FEM) are inaccurate and inefficient because the priori knowledge of
bidding strategies before any environment changes, such as load demand
variations, network congestion, and modifications of market design, is not
fully utilized. In this paper, a Bayes-adaptive Markov Decision Process in FEM
(BAMDP-FEM) is therefore developed to model the GENCOs' bidding strategy
optimization considering the priori knowledge. A novel Multi-Agent Generative
Adversarial Imitation Learning algorithm (MAGAIL-FEM) is then proposed to
enable GENCOs to learn simultaneously from priori knowledge and interactions
with changing environments. The obtained NE is a Bayesian Nash Equilibrium
(BNE) with priori knowledge transferred from the previous environment. In the
case study, the superiority of this proposed algorithm in terms of convergence
speed compared with conventional methods is verified. It is concluded that the
optimal bidding strategies in the obtained BNE can always lead to more profits
than NE due to the effective learning from the priori knowledge. Also, BNE is
more accurate and consistent with situations in real-world markets.
( 2
min )
Implicitly Normalized Forecaster (online mirror descent with Tsallis entropy
as prox-function) is known to be an optimal algorithm for adversarial
multi-armed problems (MAB). However, most of the complexity results rely on
bounded rewards or other restrictive assumptions. Recently closely related
best-of-both-worlds algorithm were proposed for both adversarial and stochastic
heavy-tailed MAB settings. This algorithm is known to be optimal in both
settings, but fails to exploit data fully. In this paper, we propose Implicitly
Normalized Forecaster with clipping for MAB problems with heavy-tailed
distribution on rewards. We derive convergence results under mild assumptions
on rewards distribution and show that the proposed method is optimal for both
linear and non-linear heavy-tailed stochastic MAB problems. Also we show that
algorithm usually performs better compared to best-of-two-worlds algorithm.
( 2
min )
Neural ordinary differential equations (neural ODEs) are a popular family of
continuous-depth deep learning models. In this work, we consider a large family
of parameterized ODEs with continuous-in-time parameters, which include
time-dependent neural ODEs. We derive a generalization bound for this class by
a Lipschitz-based argument. By leveraging the analogy between neural ODEs and
deep residual networks, our approach yields in particular a generalization
bound for a class of deep residual networks. The bound involves the magnitude
of the difference between successive weight matrices. We illustrate numerically
how this quantity affects the generalization capability of neural networks.
( 2
min )
Hyperparameter optimization (HPO) is a powerful technique for automating the
tuning of machine learning (ML) models. However, in many real-world
applications, accuracy is only one of multiple performance criteria that must
be considered. Optimizing these objectives simultaneously on a complex and
diverse search space remains a challenging task. In this paper, we propose
MO-DEHB, an effective and flexible multi-objective (MO) optimizer that extends
the recent evolutionary Hyperband method DEHB. We validate the performance of
MO-DEHB using a comprehensive suite of 15 benchmarks consisting of diverse and
challenging MO problems, including HPO, neural architecture search (NAS), and
joint NAS and HPO, with objectives including accuracy, latency and algorithmic
fairness. A comparative study against state-of-the-art MO optimizers
demonstrates that MO-DEHB clearly achieves the best performance across our 15
benchmarks.
( 2
min )
We propose a node clustering method for time-varying graphs based on the
assumption that the cluster labels are changed smoothly over time. Clustering
is one of the fundamental tasks in many science and engineering fields
including signal processing, machine learning, and data mining. Although most
existing studies focus on the clustering of nodes in static graphs, we often
encounter time-varying graphs for time-series data, e.g., social networks,
brain functional connectivity, and point clouds. In this paper, we formulate a
node clustering of time-varying graphs as an optimization problem based on
spectral clustering, with a smoothness constraint of the node labels. We solve
the problem with a primal-dual splitting algorithm. Experiments on synthetic
and real-world time-varying graphs are performed to validate the effectiveness
of the proposed approach.
( 2
min )
As a consequence of the increasing influence of machine learning on our
lives, everyone needs competencies to understand corresponding phenomena, but
also to get involved in shaping our world and making informed decisions
regarding the influences on our society. Therefore, in K-12 education, students
need to learn about core ideas and principles of machine learning. However, for
this target group, achieving all of the aforementioned goals presents an
enormous challenge. To this end, we present a teaching concept that combines a
playful and accessible unplugged approach focusing on conceptual understanding
with empowering students to actively apply machine learning methods and reflect
their influence on society, building upon decision tree learning.
( 2
min )
We study multi-agent reinforcement learning in the setting of episodic Markov
decision processes, where multiple agents cooperate via communication through a
central server. We propose a provably efficient algorithm based on value
iteration that enable asynchronous communication while ensuring the advantage
of cooperation with low communication overhead. With linear function
approximation, we prove that our algorithm enjoys an
$\tilde{\mathcal{O}}(d^{3/2}H^2\sqrt{K})$ regret with
$\tilde{\mathcal{O}}(dHM^2)$ communication complexity, where $d$ is the feature
dimension, $H$ is the horizon length, $M$ is the total number of agents, and
$K$ is the total number of episodes. We also provide a lower bound showing that
a minimal $\Omega(dM)$ communication complexity is required to improve the
performance through collaboration.
( 2
min )
As a medical doctor in Nigeria, Tobi Olatunji knows the stress of practicing in Africa’s busy hospitals. As a machine-learning scientist, he has a prescription for it. “I worked at one of West Africa’s largest hospitals, where I would routinely see more than 30 patients a day — it’s a very hard job,” said Olatunji. Read article >
( 6
min )
Make gaming a priority this GFN Thursday — time’s running out to upgrade to a GeForce NOW Priority six-month membership at 40% off the normal price. Find out how new Priority members are using the cloud to get their game on. Plus, the week brings updates for some of the hottest games in the GeForce Read article >
( 5
min )
For telcos around the world, one of the biggest challenges to upgrading networks has always been the question, “If you build it, will they come?” Asia’s leading telco, Singtel, believes the key to helping customers innovate with AI across industries — for everything from traffic and video analytics to conversational AI avatars powered by large Read article >
( 5
min )
As previous representations for reinforcement learning cannot effectively
incorporate a human-intuitive understanding of the 3D environment, they usually
suffer from sub-optimal performances. In this paper, we present Semantic-aware
Neural Radiance Fields for Reinforcement Learning (SNeRL), which jointly
optimizes semantic-aware neural radiance fields (NeRF) with a convolutional
encoder to learn 3D-aware neural implicit representation from multi-view
images. We introduce 3D semantic and distilled feature fields in parallel to
the RGB radiance fields in NeRF to learn semantic and object-centric
representation for reinforcement learning. SNeRL outperforms not only previous
pixel-based representations but also recent 3D-aware representations both in
model-free and model-based reinforcement learning.
( 2
min )
Correlation matrix visualization is essential for understanding the
relationships between variables in a dataset, but missing data can pose a
significant challenge in estimating correlation coefficients. In this paper, we
compare the effects of various missing data methods on the correlation plot,
focusing on two common missing patterns: random and monotone. We aim to provide
practical strategies and recommendations for researchers and practitioners in
creating and analyzing the correlation plot. Our experimental results suggest
that while imputation is commonly used for missing data, using imputed data for
plotting the correlation matrix may lead to a significantly misleading
inference of the relation between the features. We recommend using DPER, a
direct parameter estimation approach, for plotting the correlation matrix based
on its performance in the experiments.
( 2
min )
This paper focuses on solving a fault detection problem using multivariate
time series of vibration signals collected from planetary gearboxes in a test
rig. Various traditional machine learning and deep learning methods have been
proposed for multivariate time-series classification, including distance-based,
functional data-oriented, feature-driven, and convolution kernel-based methods.
Recent studies have shown using convolution kernel-based methods like ROCKET,
and 1D convolutional neural networks with ResNet and FCN, have robust
performance for multivariate time-series data classification. We propose an
ensemble of three convolution kernel-based methods and show its efficacy on
this fault detection problem by outperforming other approaches and achieving an
accuracy of more than 98.8\%.
( 2
min )
Likelihood-free inference methods typically make use of a distance between
simulated and real data. A common example is the maximum mean discrepancy
(MMD), which has previously been used for approximate Bayesian computation,
minimum distance estimation, generalised Bayesian inference, and within the
nonparametric learning framework. The MMD is commonly estimated at a root-$m$
rate, where $m$ is the number of simulated samples. This can lead to
significant computational challenges since a large $m$ is required to obtain an
accurate estimate, which is crucial for parameter estimation. In this paper, we
propose a novel estimator for the MMD with significantly improved sample
complexity. The estimator is particularly well suited for computationally
expensive smooth simulators with low- to mid-dimensional inputs. This claim is
supported through both theoretical results and an extensive simulation study on
benchmark simulators.
( 2
min )
In recent years neural networks have achieved impressive results on many
technological and scientific tasks. Yet, the mechanism through which these
models automatically select features, or patterns in data, for prediction
remains unclear. Identifying such a mechanism is key to advancing performance
and interpretability of neural networks and promoting reliable adoption of
these models in scientific applications. In this paper, we identify and
characterize the mechanism through which deep fully connected neural networks
learn features. We posit the Deep Neural Feature Ansatz, which states that
neural feature learning occurs by implementing the average gradient outer
product to up-weight features strongly related to model output. Our ansatz
sheds light on various deep learning phenomena including emergence of spurious
features and simplicity biases and how pruning networks can increase
performance, the "lottery ticket hypothesis." Moreover, the mechanism
identified in our work leads to a backpropagation-free method for feature
learning with any machine learning model. To demonstrate the effectiveness of
this feature learning mechanism, we use it to enable feature learning in
classical, non-feature learning models known as kernel machines and show that
the resulting models, which we refer to as Recursive Feature Machines, achieve
state-of-the-art performance on tabular data.
( 3
min )
In this paper we propose a new iterative algorithm to solve the fair PCA
(FPCA) problem. We start with the max-min fair PCA formulation originally
proposed in [1] and derive a simple and efficient iterative algorithm which is
based on the minorization-maximization (MM) approach. The proposed algorithm
relies on the relaxation of a semi-orthogonality constraint which is proved to
be tight at every iteration of the algorithm. The vanilla version of the
proposed algorithm requires solving a semi-definite program (SDP) at every
iteration, which can be further simplified to a quadratic program by
formulating the dual of the surrogate maximization problem. We also propose two
important reformulations of the fair PCA problem: a) fair robust PCA -- which
can handle outliers in the data, and b) fair sparse PCA -- which can enforce
sparsity on the estimated fair principal components. The proposed algorithms
are computationally efficient and monotonically increase their respective
design objectives at every iteration. An added feature of the proposed
algorithms is that they do not require the selection of any hyperparameter
(except for the fair sparse PCA case where a penalty parameter that controls
the sparsity has to be chosen by the user). We numerically compare the
performance of the proposed methods with two of the state-of-the-art approaches
on synthetic data sets and a real-life data set.
( 2
min )
We show the convergence of Wasserstein inverse reinforcement learning (WIRL)
for multi-objective optimizations with the projective subgradient method by
formulating an inverse problem of the optimization problem that is equivalent
to WIRL for multi-objective optimizations.
In addition, we prove convergence of inverse reinforcement learning (maximum
entropy inverse reinforcement learning, guid cost learning) for multi-objective
optimization with the projective subgradient method.
( 2
min )
We formulate a uniform tail bound for empirical processes indexed by a class
of functions, in terms of the individual deviations of the functions rather
than the worst-case deviation in the considered class. The tail bound is
established by introducing an initial "deflation" step to the standard generic
chaining argument. The resulting tail bound has a main complexity component, a
variant of Talagrand's $\gamma$ functional for the deflated function class, as
well as an instance-dependent deviation term, measured by an appropriately
scaled version of a suitable norm. Both of these terms are expressed using
certain coefficients formulated based on the relevant cumulant generating
functions. We also provide more explicit approximations for the mentioned
coefficients, when the function class lies in a given (exponential type) Orlicz
space.
( 2
min )
Robustness against image perturbations bounded by a $\ell_p$ ball have been
well-studied in recent literature. Perturbations in the real-world, however,
rarely exhibit the pixel independence that $\ell_p$ threat models assume. A
recently proposed Wasserstein distance-bounded threat model is a promising
alternative that limits the perturbation to pixel mass movements. We point out
and rectify flaws in previous definition of the Wasserstein threat model and
explore stronger attacks and defenses under our better-defined framework.
Lastly, we discuss the inability of current Wasserstein-robust models in
defending against perturbations seen in the real world. Our code and trained
models are available at https://github.com/edwardjhu/improved_wasserstein .
( 2
min )
While machine learning is currently transforming the field of histopathology,
the domain lacks a comprehensive evaluation of state-of-the-art models based on
essential but complementary quality requirements beyond a mere classification
accuracy. In order to fill this gap, we developed a new methodology to
extensively evaluate a wide range of classification models, including recent
vision transformers, and convolutional neural networks such as: ConvNeXt,
ResNet (BiT), Inception, ViT and Swin transformer, with and without supervised
or self-supervised pretraining. We thoroughly tested the models on five widely
used histopathology datasets containing whole slide images of breast, gastric,
and colorectal cancer and developed a novel approach using an image-to-image
translation model to assess the robustness of a cancer classification model
against stain variations. Further, we extended existing interpretability
methods to previously unstudied models and systematically reveal insights of
the models' classifications strategies that can be transferred to future model
architectures.
( 2
min )
This paper carries out sparse-penalized deep neural networks predictors for
learning weakly dependent processes, with a broad class of loss functions. We
deal with a general framework that includes, regression estimation,
classification, times series prediction, $\cdots$ The $\psi$-weak dependence
structure is considered, and for the specific case of bounded observations,
$\theta_\infty$-coefficients are also used. In this case of
$\theta_\infty$-weakly dependent, a non asymptotic generalization bound within
the class of deep neural networks predictors is provided. For learning both
$\psi$ and $\theta_\infty$-weakly dependent processes, oracle inequalities for
the excess risk of the sparse-penalized deep neural networks estimators are
established. When the target function is sufficiently smooth, the convergence
rate of these excess risk is close to $\mathcal{O}(n^{-1/3})$. Some simulation
results are provided, and application to the forecast of the particulate matter
in the Vit\'{o}ria metropolitan area is also considered.
( 2
min )
Creative studio Elara Systems doesn’t shy away from sensitive subjects in its work.
( 6
min )
Artificial intelligence is teaming with crowdsourcing to improve mRNA vaccines’ thermostability — the ability to avoid breaking down under heat stress — making distribution more accessible worldwide. In this episode of the NVIDIA AI Podcast, host Noah Kravitz interviews Bojan Tunguz, a physicist and senior system software engineer, and Johnny Israeli, senior manager of AI Read article >
( 5
min )
You can now register machine learning (ML) models built in Amazon SageMaker Canvas with a single click to the Amazon SageMaker Model Registry, enabling you to operationalize ML models in production. Canvas is a visual interface that enables business analysts to generate accurate ML predictions on their own—without requiring any ML experience or having to […]
( 8
min )
Today, data scientists who are training deep learning models need to identify and remediate model training issues to meet accuracy targets for production deployment, and require a way to utilize standard tools for debugging model training. Among the data scientist community, TensorBoard is a popular toolkit that allows data scientists to visualize and analyze various […]
( 8
min )
Amazon SageMaker provides a broad selection of machine learning (ML) infrastructure and model deployment options to help meet your ML inference needs. It’s a fully-managed service and integrates with MLOps tools so you can work to scale your model deployment, reduce inference costs, manage models more effectively in production, and reduce operational burden. SageMaker provides […]
( 7
min )
This is a guest post co-written with Trey Robinson, CTO at Sleepme Inc. Sleepme is an industry leader in sleep temperature management and monitoring products, including an Internet of Things (IoT) enabled sleep tracking sensor suite equipped with heart rate, respiration rate, bed and ambient temperature, humidity, and pressure sensors. Sleepme offers a smart mattress […]
( 6
min )
Understanding business trends, customer behavior, sales revenue, increase in demand, and buyer propensity all start with data. Exploring, analyzing, interpreting, and finding trends in data is essential for businesses to achieve successful outcomes. Business analysts play a pivotal role in facilitating data-driven business decisions through activities such as the visualization of business metrics and the […]
( 10
min )
Project Jupyter is a multi-stakeholder, open-source project that builds applications, open standards, and tools for data science, machine learning (ML), and computational science. The Jupyter Notebook, first released in 2011, has become a de facto standard tool used by millions of users worldwide across every possible academic, research, and industry sector. Jupyter enables users to […]
( 8
min )
Jupyter notebooks are highly favored by data scientists for their ability to interactively process data, build ML models, and test these models by making inferences on data. However, there are scenarios in which data scientists may prefer to transition from interactive development on notebooks to batch jobs. Examples of such use cases include scaling up […]
( 9
min )
Citadel founder and CEO Ken Griffin visits MIT, discusses how technology will continue to transform trading and investing.
( 9
min )
Models trained using common data-collection techniques judge rule violations more harshly than humans would, researchers report.
( 9
min )
Matt Shoulders will lead an interdisciplinary team to improve RuBisCO — the photosynthesis enzyme thought to be the holy grail for improving agricultural yield.
( 11
min )
A new computer vision system turns any shiny object into a camera of sorts, enabling an observer to see around corners or beyond obstructions.
( 10
min )
Learn 5 tips to implement cybersecurity awareness at your business and discover solutions to protect your business from cyber threats. Read on for more.
The post Building a Secure Workplace: 5 Strategies to Raise Cybersecurity Awareness appeared first on Data Science Central.
( 21
min )
Machine learning has emerged as a powerful tool for organizations across industries to enhance their operational efficiency and make data-driven decisions. With the increasing reliance of businesses on machine learning models, it is crucial to guarantee their performance as expected. At this point, monitoring the machine learning models comes into play. To put it simply,… Read More »You should never neglect to monitor your machine-learning models
The post You should never neglect to monitor your machine-learning models appeared first on Data Science Central.
( 21
min )
Amazon SageMaker Serverless Inference allows you to serve model inference requests in real time without having to explicitly provision compute instances or configure scaling policies to handle traffic variations. You can let AWS handle the undifferentiated heavy lifting of managing the underlying infrastructure and save costs in the process. A Serverless Inference endpoint spins up […]
( 13
min )
Proteins drive many biological processes, such as enzyme activity, molecular transport, and cellular support. The three-dimensional structure of a protein provides insight into its function and how it interacts with other biomolecules. Experimental methods to determine protein structure, such as X-ray crystallography and NMR spectroscopy, are expensive and time-consuming. In contrast, recently-developed computational methods can […]
( 8
min )
Healthcare data is complex and siloed, and exists in various formats. An estimated 80% of data within organizations is considered to be unstructured or “dark” data that is locked inside text, emails, PDFs, and scanned documents. This data is difficult to interpret or analyze programmatically and limits how organizations can derive insights from it and […]
( 7
min )
Amazon SageMaker provides a number of options for users who are looking for a solution to host their machine learning (ML) models. Of these options, one of the key features that SageMaker provides is real-time inference. Real-time inference workloads can have varying levels of requirements and service level agreements (SLAs) in terms of latency and […]
( 15
min )
3D artist Milan Dey finds inspiration in games, movies, comics and pop culture. He drew from all of the above when creating a stunning 3D scene of Mayan ruins, The Hidden Temple of Itzamná, this week In the NVIDIA Studio.
( 7
min )
Researchers identify a property that helps computer vision models learn to represent the visual world in a more stable, predictable way.
( 10
min )
With Amazon SageMaker, you can manage the whole end-to-end machine learning (ML) lifecycle. It offers many native capabilities to help manage ML workflows aspects, such as experiment tracking, and model governance via the model registry. This post provides a solution tailored to customers that are already using MLflow, an open-source platform for managing ML workflows. […]
( 15
min )
Sometimes it can be very beneficial to use tools such as compilers that can modify and compile your models for optimal inference performance. In this post, we explore TensorRT and how to use it with Amazon SageMaker inference using NVIDIA Triton Inference Server. We explore how TensorRT works and how to host and optimize these […]
( 15
min )
Warning: very long, 2-part blog series. But this topic is too important to not carefully explain how we can educate and empower everyone to participate in the AI conversation. Our success as a society depends upon our ability to include everyone in this conversation. “I love it when a plan comes together” – Hannibal Smith,… Read More »AI for Everyone: Learn How to Think Like a Data Scientist – Part 1
The post AI for Everyone: Learn How to Think Like a Data Scientist – Part 1 appeared first on Data Science Central.
( 22
min )
Machine learning in Healthcare industry.
The post How Machine Learning is Revolutionizing the Healthcare Industry appeared first on Data Science Central.
( 21
min )
Safety has been a critical issue for the deployment of learning-based
approaches in real-world applications. To address this issue, control barrier
function (CBF) and its variants have attracted extensive attention for
safety-critical control. However, due to the myopic one-step nature of CBF and
the lack of principled methods to design the class-$\mathcal{K}$ functions,
there are still fundamental limitations of current CBFs: optimality, stability,
and feasibility. In this paper, we proposed a novel and unified approach to
address these limitations with Adaptive Multi-step Control Barrier Function
(AM-CBF), where we parameterize the class-$\mathcal{K}$ function by a neural
network and train it together with the reinforcement learning policy. Moreover,
to mitigate the myopic nature, we propose a novel \textit{multi-step training
and single-step execution} paradigm to make CBF farsighted while the execution
remains solving a single-step convex quadratic program. Our method is evaluated
on the first and second-order systems in various scenarios, where our approach
outperforms the conventional CBF both qualitatively and quantitatively.
( 2
min )
Small molecules in biological samples are studied to provide information
about disease states, environmental toxins, natural product drug discovery, and
many other applications. The primary window into the composition of small
molecule mixtures is tandem mass spectrometry (MS2), which produces data that
are of high sensitivity and part per million resolution. We adopt multi-scale
sinusoidal embeddings of the mass data in MS2 designed to meet the challenge of
learning from the full resolution of MS2 data. Using these embeddings, we
provide a new state of the art model for spectral library search, the standard
task for initial evaluation of MS2 data. We also introduce a new task, chemical
property prediction from MS2 data, that has natural applications in
high-throughput MS2 experiments and show that an average $R^2$ of 80\% for
novel compounds can be achieved across 10 chemical properties prioritized by
medicinal chemists. We use dimensionality reduction techniques and experiments
with different floating point resolutions to show the essential role
multi-scale sinusoidal embeddings play in learning from MS2 data.
( 2
min )
Motivated by the recent success of Machine Learning tools in wireless
communications, the idea of semantic communication by Weaver from 1949 has
gained attention. It breaks with Shannon's classic design paradigm by aiming to
transmit the meaning, i.e., semantics, of a message instead of its exact
version, allowing for information rate savings. In this work, we apply the
Stochastic Policy Gradient (SPG) to design a semantic communication system by
reinforcement learning, not requiring a known or differentiable channel model -
a crucial step towards deployment in practice. Further, we motivate the use of
SPG for both classic and semantic communication from the maximization of the
mutual information between received and target variables. Numerical results
show that our approach achieves comparable performance to a model-aware
approach based on the reparametrization trick, albeit with a decreased
convergence rate.
( 2
min )
This paper considers the problem of helping humans exercise scalable
oversight over deep neural networks (DNNs). Adversarial examples can be useful
by helping to reveal weaknesses in DNNs, but they can be difficult to interpret
or draw actionable conclusions from. Some previous works have proposed using
human-interpretable adversarial attacks including copy/paste attacks in which
one natural image pasted into another causes an unexpected misclassification.
We build on these with two contributions. First, we introduce Search for
Natural Adversarial Features Using Embeddings (SNAFUE) which offers a fully
automated method for finding copy/paste attacks. Second, we use SNAFUE to red
team an ImageNet classifier. We reproduce copy/paste attacks from previous
works and find hundreds of other easily-describable vulnerabilities, all
without a human in the loop. Code is available at
https://github.com/thestephencasper/snafue
( 2
min )
We propose a novel graph-regularized neural network (GRNN) algorithm for tree
species classification. The proposed algorithm encompasses superpixel-based
segmentation for graph construction, a pixel-wise neural network classifier,
and the label propagation technique to generate an accurate and realistic
(emulating tree crowns) classification map on a sparsely annotated data set.
GRNN outperforms several state-of-the-art techniques not only for the standard
Indian Pines HSI but also achieves a high classification accuracy (approx. 92%)
on a new HSI data set collected over the heterogeneous forests of French Guiana
(FG) when less than 1% of the pixels are labeled. We further show that GRNN is
competitive with the state-of-the-art semi-supervised methods and exhibits a
small deviation in accuracy for different numbers of training samples and over
repeated trials with randomly sampled labeled pixels for training.
( 2
min )
Bayesian hierarchical mixture clustering (BHMC) improves traditionalBayesian
hierarchical clustering by replacing conventional Gaussian-to-Gaussian kernels
with a Hierarchical Dirichlet Process Mixture Model(HDPMM) for parent-to-child
diffusion in the generative process. However,BHMC may produce trees with high
nodal variance, indicating weak separation between nodes at higher levels. To
address this issue, we employ Posterior Regularization, which imposes
max-margin constraints on nodes at every level to enhance cluster separation.
We illustrate how to apply PR toBHMC and demonstrate its effectiveness in
improving the BHMC model.
( 2
min )
Strong demand for autonomous vehicles and the wide availability of 3D sensors
are continuously fueling the proposal of novel methods for 3D object detection.
In this paper, we provide a comprehensive survey of recent developments from
2012-2021 in 3D object detection covering the full pipeline from input data,
over data representation and feature extraction to the actual detection
modules. We introduce fundamental concepts, focus on a broad range of different
approaches that have emerged over the past decade, and propose a
systematization that provides a practical framework for comparing these
approaches with the goal of guiding future development, evaluation and
application activities. Specifically, our survey and systematization of 3D
object detection models and methods can help researchers and practitioners to
get a quick overview of the field by decomposing 3DOD solutions into more
manageable pieces.
( 2
min )
Photoplethysmogram (PPG) signals are easily contaminated by motion artifacts
in real-world settings, despite their widespread use in Internet-of-Things
(IoT) based wearable and smart health devices for cardiovascular health
monitoring. This study proposed a lightweight deep neural network, called
Tiny-PPG, for accurate and real-time PPG artifact segmentation on IoT edge
devices. The model was trained and tested on a public dataset, PPG DaLiA, which
featured complex artifacts with diverse lengths and morphologies during various
daily activities of 15 subjects using a watch-type device (Empatica E4). The
model structure, training method and loss function were specifically designed
to balance detection accuracy and speed for real-time PPG artifact detection in
resource-constrained embedded devices. To optimize the model size and
capability in multi-scale feature representation, the model employed deep
separable convolution and atrous spatial pyramid pooling modules, respectively.
Additionally, the contrastive loss was also utilized to further optimize the
feature embeddings. With additional model pruning, Tiny-PPG achieved
state-of-the-art detection accuracy of 87.8% while only having 19,726 model
parameters (0.15 megabytes), and was successfully deployed on an STM32 embedded
system for real-time PPG artifact detection. Therefore, this study provides an
effective solution for resource-constraint IoT smart health devices in PPG
artifact detection.
( 2
min )
Natural language generation from structured data mainly focuses on
surface-level descriptions, suffering from uncontrollable content selection and
low fidelity. Previous works leverage logical forms to facilitate logical
knowledge-conditioned text generation. Though achieving remarkable progress,
they are data-hungry, which makes the adoption for real-world applications
challenging with limited data. To this end, this paper proposes a unified
framework for logical knowledge-conditioned text generation in the few-shot
setting. With only a few seeds logical forms (e.g., 20/100 shot), our approach
leverages self-training and samples pseudo logical forms based on content and
structure consistency. Experimental results demonstrate that our approach can
obtain better few-shot performance than baselines.
( 2
min )
Latent variable modeling in non-autoregressive neural machine translation
(NAT) is a promising approach to mitigate the multimodality problem. In the
previous works, they added an auxiliary model to estimate the posterior
distribution of the latent variable conditioned on the source and target
sentences. However, it causes several disadvantages, such as redundant
information extraction in the latent variable, increasing parameters, and a
tendency to ignore a part of the information from the inputs. In this paper, we
propose a new latent variable modeling that is based on a dual reconstruction
perspective and an advanced hierarchical latent modeling approach. Our proposed
method, {\em LadderNMT}, shares a latent space across both languages so that it
hypothetically alleviates or solves the above disadvantages. Experimental
results quantitatively and qualitatively demonstrate that our proposed latent
variable modeling learns an advantageous latent space and significantly
improves translation quality in WMT translation tasks.
( 2
min )
Achieving resource efficiency while preserving end-user experience is
non-trivial for cloud application operators. As cloud applications
progressively adopt microservices, resource managers are faced with two
distinct levels of system behavior: the end-to-end application latency and
per-service resource usage. Translation between these two levels, however, is
challenging because user requests traverse heterogeneous services that
collectively (but unevenly) contribute to the end-to-end latency. This paper
presents Autothrottle, a bi-level learning-assisted resource management
framework for SLO-targeted microservices. It architecturally decouples
mechanisms of application SLO feedback and service resource control, and
bridges them with the notion of performance targets. This decoupling enables
targeted control policies for these two mechanisms, where we combine
lightweight heuristics and learning techniques. We evaluate Autothrottle on
three microservice applications, with workload traces from production
scenarios. Results show its superior CPU resource saving, up to 26.21% over the
best-performing baseline, and up to 93.84% over all baselines.
( 2
min )
This document presents some early explorations of applying Softly Masked
Language Modelling (SMLM) to symbolic music generation. SMLM can be seen as a
generalisation of masked language modelling (MLM), where instead of each
element of the input set being either known or unknown, elements can be partly
known. We demonstrate some results of applying SMLM to constrained symbolic
music generation using a transformer encoder architecture. Several audio
examples are available at https://erl-j.github.io/smlm-web-supplement/
( 2
min )
ChatGPT is another large language model (LLM) inline but due to its
performance and ability to converse effectively, it has gained a huge
popularity amongst research as well as industrial community. Recently, many
studies have been published to show the effectiveness, efficiency, integration,
and sentiments of chatGPT and other LLMs. In contrast, this study focuses on
the important aspects that are mostly overlooked, i.e. sustainability, privacy,
digital divide, and ethics and suggests that not only chatGPT but every
subsequent entry in the category of conversational bots should undergo
Sustainability, PrivAcy, Digital divide, and Ethics (SPADE) evaluation. This
paper discusses in detail about the issues and concerns raised over chatGPT in
line with aforementioned characteristics. We support our hypothesis by some
preliminary data collection and visualizations along with hypothesized facts.
We also suggest mitigations and recommendations for each of the concerns.
Furthermore, we also suggest some policies and recommendations for AI policy
act, if designed by the governments.
( 2
min )
In recent years, multitudes of researches have applied deep learning to
automatic sleep stage classification. Whereas actually, these works have paid
less attention to the issue of cross-subject in sleep staging. At the same
time, emerging neuroscience theories on inter-subject correlations can provide
new insights for cross-subject analysis. This paper presents the MViTime model
that have been used in sleep staging study. And we implement the inter-subject
correlation theory through contrastive learning, providing a feasible solution
to address the cross-subject problem in sleep stage classification. Finally,
experimental results and conclusions are presented, demonstrating that the
developed method has achieved state-of-the-art performance on sleep staging.
The results of the ablation experiment also demonstrate the effectiveness of
the cross-subject approach based on contrastive learning.
( 2
min )
We use explainable neural networks to connect the evolutionary history of
dark matter halos with their density profiles. The network captures independent
factors of variation in the density profiles within a low-dimensional
representation, which we physically interpret using mutual information. Without
any prior knowledge of the halos' evolution, the network recovers the known
relation between the early time assembly and the inner profile, and discovers
that the profile beyond the virial radius is described by a single parameter
capturing the most recent mass accretion rate. The results illustrate the
potential for machine-assisted scientific discovery in complicated
astrophysical datasets.
( 2
min )
Selecting a minimal feature set that is maximally informative about a target
variable is a central task in machine learning and statistics. Information
theory provides a powerful framework for formulating feature selection
algorithms -- yet, a rigorous, information-theoretic definition of feature
relevancy, which accounts for feature interactions such as redundant and
synergistic contributions, is still missing. We argue that this lack is
inherent to classical information theory which does not provide measures to
decompose the information a set of variables provides about a target into
unique, redundant, and synergistic contributions. Such a decomposition has been
introduced only recently by the partial information decomposition (PID)
framework. Using PID, we clarify why feature selection is a conceptually
difficult problem when approached using information theory and provide a novel
definition of feature relevancy and redundancy in PID terms. From this
definition, we show that the conditional mutual information (CMI) maximizes
relevancy while minimizing redundancy and propose an iterative, CMI-based
algorithm for practical feature selection. We demonstrate the power of our
CMI-based algorithm in comparison to the unconditional mutual information on
benchmark examples and provide corresponding PID estimates to highlight how PID
allows to quantify information contribution of features and their interactions
in feature-selection problems.
( 3
min )
Generating explanations for reinforcement learning (RL) is challenging as
actions may produce long-term effects on the future. In this paper, we develop
a novel framework for explainable RL by learning a causal world model without
prior knowledge of the causal structure of the environment. The model captures
the influence of actions, allowing us to interpret the long-term effects of
actions through causal chains, which present how actions influence
environmental variables and finally lead to rewards. Different from most
explanatory models which suffer from low accuracy, our model remains accurate
while improving explainability, making it applicable in model-based learning.
As a result, we demonstrate that our causal model can serve as the bridge
between explainability and learning.
( 2
min )
Since its inception, the field of unbiased learning to rank (ULTR) has
remained very active and has seen several impactful advancements in recent
years. This tutorial provides both an introduction to the core concepts of the
field and an overview of recent advancements in its foundations along with
several applications of its methods. The tutorial is divided into four parts:
Firstly, we give an overview of the different forms of bias that can be
addressed with ULTR methods. Secondly, we present a comprehensive discussion of
the latest estimation techniques in the ULTR field. Thirdly, we survey
published results of ULTR in real-world applications. Fourthly, we discuss the
connection between ULTR and fairness in ranking. We end by briefly reflecting
on the future of ULTR research and its applications. This tutorial is intended
to benefit both researchers and industry practitioners who are interested in
developing new ULTR solutions or utilizing them in real-world applications.
( 2
min )
Detecting plagiarism involves finding similar items in two different sources.
In this article, we propose a novel method for detecting plagiarism that is
based on attention mechanism-based long short-term memory (LSTM) and
bidirectional encoder representations from transformers (BERT) word embedding,
enhanced with optimized differential evolution (DE) method for pre-training and
a focal loss function for training. BERT could be included in a downstream task
and fine-tuned as a task-specific BERT can be included in a downstream task and
fine-tuned as a task-specific structure, while the trained BERT model is
capable of detecting various linguistic characteristics. Unbalanced
classification is one of the primary issues with plagiarism detection. We
suggest a focal loss-based training technique that carefully learns minority
class instances to solve this. Another issue that we tackle is the training
phase itself, which typically employs gradient-based methods like
back-propagation for the learning process and thus suffers from some drawbacks,
including sensitivity to initialization. To initiate the BP process, we suggest
a novel DE algorithm that makes use of a clustering-based mutation operator.
Here, a winning cluster is identified for the current DE population, and a
fresh updating method is used to produce potential answers. We evaluate our
proposed approach on three benchmark datasets ( MSRP, SNLI, and SemEval2014)
and demonstrate that it performs well when compared to both conventional and
population-based methods.
( 3
min )
This work investigates pretrained audio representations for few shot Sound
Event Detection. We specifically address the task of few shot detection of
novel acoustic sequences, or sound events with semantically meaningful temporal
structure, without assuming access to non-target audio. We develop procedures
for pretraining suitable representations, and methods which transfer them to
our few shot learning scenario. Our experiments evaluate the general purpose
utility of our pretrained representations on AudioSet, and the utility of
proposed few shot methods via tasks constructed from real-world acoustic
sequences. Our pretrained embeddings are suitable to the proposed task, and
enable multiple aspects of our few shot framework.
( 2
min )
Recent years have witnessed the proliferation of traffic accidents, which led
wide researches on Automated Vehicle (AV) technologies to reduce vehicle
accidents, especially on risk assessment framework of AV technologies. However,
existing time-based frameworks can not handle complex traffic scenarios and
ignore the motion tendency influence of each moving objects on the risk
distribution, leading to performance degradation. To address this problem, we
novelly propose a comprehensive driving risk management framework named RCP-RF
based on potential field theory under Connected and Automated Vehicles (CAV)
environment, where the pedestrian risk metric are combined into a unified
road-vehicle driving risk management framework. Different from existing
algorithms, the motion tendency between ego and obstacle cars and the
pedestrian factor are legitimately considered in the proposed framework, which
can improve the performance of the driving risk model. Moreover, it requires
only O(N 2) of time complexity in the proposed method. Empirical studies
validate the superiority of our proposed framework against state-of-the-art
methods on real-world dataset NGSIM and real AV platform.
( 2
min )
We study principal component analysis (PCA), where given a dataset in
$\mathbb{R}^d$ from a distribution, the task is to find a unit vector $v$ that
approximately maximizes the variance of the distribution after being projected
along $v$. Despite being a classical task, standard estimators fail drastically
if the data contains even a small fraction of outliers, motivating the problem
of robust PCA. Recent work has developed computationally-efficient algorithms
for robust PCA that either take super-linear time or have sub-optimal error
guarantees. Our main contribution is to develop a nearly-linear time algorithm
for robust PCA with near-optimal error guarantees. We also develop a
single-pass streaming algorithm for robust PCA with memory usage nearly-linear
in the dimension.
( 2
min )
Analysis of Electrochemical Impedance Spectroscopy (EIS) data for
electrochemical systems often consists of defining an Equivalent Circuit Model
(ECM) using expert knowledge and then optimizing the model parameters to
deconvolute various resistance, capacitive, inductive, or diffusion responses.
For small data sets, this procedure can be conducted manually; however, it is
not feasible to manually define a proper ECM for extensive data sets with a
wide range of EIS responses. Automatic identification of an ECM would
substantially accelerate the analysis of large sets of EIS data. We showcase
machine learning methods to classify the ECMs of 9,300 impedance spectra
provided by QuantumScape for the BatteryDEV hackathon. The best-performing
approach is a gradient-boosted tree model utilizing a library to automatically
generate features, followed by a random forest model using the raw spectral
data. A convolutional neural network using boolean images of Nyquist
representations is presented as an alternative, although it achieves a lower
accuracy. We publish the data and open source the associated code. The
approaches described in this article can serve as benchmarks for further
studies. A key remaining challenge is the identifiability of the labels,
underlined by the model performances and the comparison of misclassified
spectra.
( 3
min )
We provide a psychometric-grounded exposition of bias and fairness as applied
to a typical machine learning pipeline for affective computing. We expand on an
interpersonal communication framework to elucidate how to identify sources of
bias that may arise in the process of inferring human emotions and other
psychological constructs from observed behavior. Various methods and metrics
for measuring fairness and bias are discussed along with pertinent implications
within the United States legal context. We illustrate how to measure some types
of bias and fairness in a case study involving automatic personality and
hireability inference from multimodal data collected in video interviews for
mock job applications. We encourage affective computing researchers and
practitioners to encapsulate bias and fairness in their research processes and
products and to consider their role, agency, and responsibility in promoting
equitable and just systems.
( 2
min )
This paper evaluates the viability of using fixed language models for
training text classification networks on low-end hardware. We combine language
models with a CNN architecture and put together a comprehensive benchmark with
8 datasets covering single-label and multi-label classification of topic,
sentiment, and genre. Our observations are distilled into a list of trade-offs,
concluding that there are scenarios, where not fine-tuning a language model
yields competitive effectiveness at faster training, requiring only a quarter
of the memory compared to fine-tuning.
( 2
min )
In this work, we propose a novel evolutionary algorithm for neural
architecture search, applicable to global search spaces. The algorithm's
architectural representation organizes the topology in multiple hierarchical
modules, while the design process exploits this representation, in order to
explore the search space. We also employ a curation system, which promotes the
utilization of well performing sub-structures to subsequent generations. We
apply our method to Fashion-MNIST and NAS-Bench101, achieving accuracies of
$93.2\%$ and $94.8\%$ respectively in a relatively small number of generations.
( 2
min )
Many machine learning (ML) libraries are accessible online for ML
practitioners. Typical ML pipelines are complex and consist of a series of
steps, each of them invoking several ML libraries. In this demo paper, we
present ExeKGLib, a Python library that allows users with coding skills and
minimal ML knowledge to build ML pipelines. ExeKGLib relies on knowledge graphs
to improve the transparency and reusability of the built ML workflows, and to
ensure that they are executable. We demonstrate the usage of ExeKGLib and
compare it with conventional ML code to show its benefits.
( 2
min )
Numerical models are used widely for parameter reconstructions in the field
of optical nano metrology. To obtain geometrical parameters of a nano
structured line grating, we fit a finite element numerical model to an
experimental data set by using the Bayesian target vector optimization method.
Gaussian process surrogate models are trained during the reconstruction.
Afterwards, we employ a Markov chain Monte Carlo sampler on the surrogate
models to determine the full model parameter distribution for the reconstructed
model parameters. The choice of numerical discretization parameters, like the
polynomial order of the finite element ansatz functions, impacts the numerical
discretization error of the forward model. In this study we investigate the
impact of numerical discretization parameters of the forward problem on the
reconstructed parameters as well as on the model parameter distributions. We
show that such a convergence study allows to determine numerical parameters
which allow for efficient and accurate reconstruction results.
( 2
min )
We describe how interpretable boosting algorithms based on ridge-regularized
generalized linear models can be used to analyze high-dimensional environmental
data. We illustrate this by using environmental, social, human and biophysical
data to predict the financial vulnerability of farmers in Chile and Tunisia
against climate hazards. We show how group structures can be considered and how
interactions can be found in high-dimensional datasets using a novel 2-step
boosting approach. The advantages and efficacy of the proposed method are shown
and discussed. Results indicate that the presence of interaction effects only
improves predictive power when included in two-step boosting. The most
important variable in predicting all types of vulnerabilities are natural
assets. Other important variables are the type of irrigation, economic assets
and the presence of crop damage of near farms.
( 2
min )
Random graphs are increasingly becoming objects of interest for modeling
networks in a wide range of applications. Latent position random graph models
posit that each node is associated with a latent position vector, and that
these vectors follow some geometric structure in the latent space. In this
paper, we consider random dot product graphs, in which an edge is formed
between two nodes with probability given by the inner product of their
respective latent positions. We assume that the latent position vectors lie on
an unknown one-dimensional curve and are coupled with a response covariate via
a regression model. Using the geometry of the underlying latent position
vectors, we propose a manifold learning and graph embedding technique to
predict the response variable on out-of-sample nodes, and we establish
convergence guarantees for these responses. Our theoretical results are
supported by simulations and an application to Drosophila brain data.
( 2
min )
This paper formulates a general cross validation framework for signal
denoising. The general framework is then applied to nonparametric regression
methods such as Trend Filtering and Dyadic CART. The resulting cross validated
versions are then shown to attain nearly the same rates of convergence as are
known for the optimally tuned analogues. There did not exist any previous
theoretical analyses of cross validated versions of Trend Filtering or Dyadic
CART. To illustrate the generality of the framework we also propose and study
cross validated versions of two fundamental estimators; lasso for high
dimensional linear regression and singular value thresholding for matrix
estimation. Our general framework is inspired by the ideas in Chatterjee and
Jafarov (2015) and is potentially applicable to a wide range of estimation
methods which use tuning parameters.
( 2
min )
This paper provide several mathematical analyses of the diffusion model in
machine learning. The drift term of the backwards sampling process is
represented as a conditional expectation involving the data distribution and
the forward diffusion. The training process aims to find such a drift function
by minimizing the mean-squared residue related to the conditional expectation.
Using small-time approximations of the Green's function of the forward
diffusion, we show that the analytical mean drift function in DDPM and the
score function in SGM asymptotically blow up in the final stages of the
sampling process for singular data distributions such as those concentrated on
lower-dimensional manifolds, and is therefore difficult to approximate by a
network. To overcome this difficulty, we derive a new target function and
associated loss, which remains bounded even for singular data distributions. We
illustrate the theoretical findings with several numerical examples.
( 2
min )
We study principal component analysis (PCA), where given a dataset in
$\mathbb{R}^d$ from a distribution, the task is to find a unit vector $v$ that
approximately maximizes the variance of the distribution after being projected
along $v$. Despite being a classical task, standard estimators fail drastically
if the data contains even a small fraction of outliers, motivating the problem
of robust PCA. Recent work has developed computationally-efficient algorithms
for robust PCA that either take super-linear time or have sub-optimal error
guarantees. Our main contribution is to develop a nearly-linear time algorithm
for robust PCA with near-optimal error guarantees. We also develop a
single-pass streaming algorithm for robust PCA with memory usage nearly-linear
in the dimension.
( 2
min )
Clustering is at the very core of machine learning, and its applications
proliferate with the increasing availability of data. However, as datasets
grow, comparing clusterings with an adjustment for chance becomes
computationally difficult, preventing unbiased ground-truth comparisons and
solution selection. We propose FastAMI, a Monte Carlo-based method to
efficiently approximate the Adjusted Mutual Information (AMI) and extend it to
the Standardized Mutual Information (SMI). The approach is compared with the
exact calculation and a recently developed variant of the AMI based on pairwise
permutations, using both synthetic and real data. In contrast to the exact
calculation our method is fast enough to enable these adjusted
information-theoretic comparisons for large datasets while maintaining
considerably more accurate results than the pairwise approach.
( 2
min )
The analysis of large-scale time-series network data, such as social media
and email communications, remains a significant challenge for graph analysis
methodology. In particular, the scalability of graph analysis is a critical
issue hindering further progress in large-scale downstream inference. In this
paper, we introduce a novel approach called "temporal encoder embedding" that
can efficiently embed large amounts of graph data with linear complexity. We
apply this method to an anonymized time-series communication network from a
large organization spanning 2019-2020, consisting of over 100 thousand vertices
and 80 million edges. Our method embeds the data within 10 seconds on a
standard computer and enables the detection of communication pattern shifts for
individual vertices, vertex communities, and the overall graph structure.
Through supporting theory and synthesis studies, we demonstrate the theoretical
soundness of our approach under random graph models and its numerical
effectiveness through simulation studies.
( 2
min )
Jeff Wilke SM '93, former CEO of Amazon’s Worldwide Consumer business, brings his LGO playbook to his new mission of revitalizing manufacturing in the U.S.
( 12
min )
In this post, we discuss a machine learning (ML) solution for complex image searches using Amazon Kendra and Amazon Rekognition. Specifically, we use the example of architecture diagrams for complex images due to their incorporation of numerous different visual icons and text. With the internet, searching and obtaining an image has never been easier. Most […]
( 17
min )
This is a joint post co-written by AWS and Voxel51. Voxel51 is the company behind FiftyOne, the open-source toolkit for building high-quality datasets and computer vision models. A retail company is building a mobile app to help customers buy clothes. To create this app, they need a high-quality dataset containing clothing images, labeled with different […]
( 16
min )
AI Weirdness: the strange side of machine learning
( 2
min )
Automated revenue cycle management (RCM) is becoming an increasingly vital component in the healthcare industry, streamlining and accurately processing complex billing tasks. Using artificial intelligence (AI) and machine learning capabilities will help in ensuring accuracy, minimize human error, and free up personnel to focus on more important tasks. In addition, utilizing AI for RCM will… Read More »Maximizing Revenue in Psychology Practices: Leveraging AI for Billing Optimization
The post Maximizing Revenue in Psychology Practices: Leveraging AI for Billing Optimization appeared first on Data Science Central.
( 21
min )
The world of artificial intelligence (AI) and machine learning (ML) has been witnessing a paradigm shift with the rise of generative AI models that can create human-like text, images, code, and audio. Compared to classical ML models, generative AI models are significantly bigger and more complex. However, their increasing complexity also comes with high costs […]
( 12
min )
Time series forecasting refers to the process of predicting future values of time series data (data that is collected at regular intervals over time). Simple methods for time series forecasting use historical values of the same variable whose future values need to be predicted, whereas more complex, machine learning (ML)-based methods use additional information, such […]
( 16
min )
Generative AI is gaining a lot of public attention at present, with talk around products such as GPT4, ChatGPT, DALL-E2, Bard, and many other AI technologies. Many customers have been asking for more information on AWS’s generative AI solutions. The aim of this post is to address those needs. This post provides an overview of […]
( 10
min )
Diffusion models have been used to generate photorealistic images and short videos, compose music, and synthesize speech. In a new paper, Microsoft Researchers explore how they can be used to imitate human behavior in interactive environments.
The post Using generative AI to imitate human behavior appeared first on Microsoft Research.
( 11
min )
Diffusion models have been used to generate photorealistic images and short videos, compose music, and synthesize speech. In a new paper, Microsoft Researchers explore how they can be used to imitate human behavior in interactive environments.
The post Using generative AI to imitate human behavior appeared first on Microsoft Research.
( 11
min )
Kris Kersey is an embedded software developer with over 20 years of experience, an educational YouTuber with 30,000+ subscribers, and a lifelong lover of comics and cosplay. These interests and expertise came together in his first-ever project using the NVIDIA Jetson platform for edge AI and robotics when he created a fully functional superhero helmet Read article >
( 6
min )
What has it got in its pocketses? More games coming in May, that’s what. GFN Thursday gets the summer started early with two newly supported games this week and 16 more coming later this month — including The Lord of the Rings: Gollum. Don’t forget to take advantage of the limited-time discount on six-month Priority Read article >
( 6
min )
The system they developed eliminates a source of bias in simulations, leading to improved algorithms that can boost the performance of applications.
( 9
min )
eXplainable artificial intelligence (XAI) methods have emerged to convert the
black box of machine learning models into a more digestible form. These methods
help to communicate how the model works with the aim of making machine learning
models more transparent and increasing the trust of end-users into their
output. SHapley Additive exPlanations (SHAP) and Local Interpretable Model
Agnostic Explanation (LIME) are two widely used XAI methods particularly with
tabular data. In this commentary piece, we discuss the way the explainability
metrics of these two methods are generated and propose a framework for
interpretation of their outputs, highlighting their weaknesses and strengths.
( 2
min )
This paper describes our submission to the MEDIQA-Chat 2023 shared task for
automatic clinical note generation from doctor-patient conversations. We report
results for two approaches: the first fine-tunes a pre-trained language model
(PLM) on the shared task data, and the second uses few-shot in-context learning
(ICL) with a large language model (LLM). Both achieve high performance as
measured by automatic metrics (e.g. ROUGE, BERTScore) and ranked second and
first, respectively, of all submissions to the shared task. Expert human
scrutiny indicates that notes generated via the ICL-based approach with GPT-4
are preferred about as often as human-written notes, making it a promising path
toward automated note generation from doctor-patient conversations.
( 2
min )
Contrastively trained encoders have recently been proven to invert the
data-generating process: they encode each input, e.g., an image, into the true
latent vector that generated the image (Zimmermann et al., 2021). However,
real-world observations often have inherent ambiguities. For instance, images
may be blurred or only show a 2D view of a 3D object, so multiple latents could
have generated them. This makes the true posterior for the latent vector
probabilistic with heteroscedastic uncertainty. In this setup, we extend the
common InfoNCE objective and encoders to predict latent distributions instead
of points. We prove that these distributions recover the correct posteriors of
the data-generating process, including its level of aleatoric uncertainty, up
to a rotation of the latent space. In addition to providing calibrated
uncertainty estimates, these posteriors allow the computation of credible
intervals in image retrieval. They comprise images with the same latent as a
given query, subject to its uncertainty. Code is available at
https://github.com/mkirchhof/Probabilistic_Contrastive_Learning
( 2
min )
Differentiable particle filters are an emerging class of particle filtering
methods that use neural networks to construct and learn parametric state-space
models. In real-world applications, both the state dynamics and measurements
can switch between a set of candidate models. For instance, in target tracking,
vehicles can idle, move through traffic, or cruise on motorways, and
measurements are collected in different geographical or weather conditions.
This paper proposes a new differentiable particle filter for regime-switching
state-space models. The method can learn a set of unknown candidate dynamic and
measurement models and track the state posteriors. We evaluate the performance
of the novel algorithm in relevant models, showing its great performance
compared to other competitive algorithms.
( 2
min )
We present a new method for functional tissue unit segmentation at the
cellular level, which utilizes the latest deep learning semantic segmentation
approaches together with domain adaptation and semi-supervised learning
techniques. This approach allows for minimizing the domain gap, class
imbalance, and captures settings influence between HPA and HubMAP datasets. The
presented approach achieves comparable with state-of-the-art-result in
functional tissue unit segmentation at the cellular level. The source code is
available at https://github.com/VSydorskyy/hubmap_2022_htt_solution
( 2
min )
Prior research has investigated the impact of various linguistic features on
cross-lingual transfer performance. In this study, we investigate the manner in
which this effect can be mapped onto the representation space. While past
studies have focused on the impact on cross-lingual alignment in multilingual
language models during fine-tuning, this study examines the absolute evolution
of the respective language representation spaces produced by MLLMs. We place a
specific emphasis on the role of linguistic characteristics and investigate
their inter-correlation with the impact on representation spaces and
cross-lingual transfer performance. Additionally, this paper provides
preliminary evidence of how these findings can be leveraged to enhance transfer
to linguistically distant languages.
( 2
min )
We present a study using new computational methods, based on a novel
combination of machine learning for inferring admixture hidden Markov models
and probabilistic model checking, to uncover interaction styles in a mobile
app. These styles are then used to inform a redesign, which is implemented,
deployed, and then analysed using the same methods. The data sets are logged
user traces, collected over two six-month deployments of each version,
involving thousands of users and segmented into different time intervals. The
methods do not assume tasks or absolute metrics such as measures of engagement,
but uncover the styles through unsupervised inference of clusters and analysis
with probabilistic temporal logic. For both versions there was a clear
distinction between the styles adopted by users during the first day/week/month
of usage, and during the second and third months, a result we had not
anticipated.
( 2
min )
Data valuation using Shapley value has emerged as a prevalent research domain
in machine learning applications. However, it is a challenge to address the
role of order in data cooperation as most research lacks such discussion. To
tackle this problem, this paper studies the definition of the partial ordinal
Shapley value by group theory in abstract algebra. Besides, since the
calculation of the partial ordinal Shapley value requires exponential time,
this paper also gives three algorithms for approximating the results. The
Truncated Monte Carlo algorithm is derived from the classic Shapley value
approximation algorithm. The Classification Monte Carlo algorithm and the
Classification Truncated Monte Carlo algorithm are based on the fact that the
data points in the same class provide similar information, then we can
accelerate the calculation by leaving out some data points in each class.
( 2
min )
Respiratory syncytial virus (RSV) is one of the most dangerous respiratory
diseases for infants and young children. Due to the nonpharmaceutical
intervention (NPI) imposed in the COVID-19 outbreak, the seasonal transmission
pattern of RSV has been discontinued in 2020 and then shifted months ahead in
2021 in the northern hemisphere. It is critical to understand how COVID-19
impacts RSV and build predictive algorithms to forecast the timing and
intensity of RSV reemergence in post-COVID-19 seasons. In this paper, we
propose a deep coupled tensor factorization machine, dubbed as DeCom, for post
COVID-19 RSV prediction. DeCom leverages tensor factorization and residual
modeling. It enables us to learn the disrupted RSV transmission reliably under
COVID-19 by taking both the regular seasonal RSV transmission pattern and the
NPI into consideration. Experimental results on a real RSV dataset show that
DeCom is more accurate than the state-of-the-art RSV prediction algorithms and
achieves up to 46% lower root mean square error and 49% lower mean absolute
error for country-level prediction compared to the baselines.
( 2
min )
In an effort to address the training instabilities of GANs, we introduce a
class of dual-objective GANs with different value functions (objectives) for
the generator (G) and discriminator (D). In particular, we model each objective
using $\alpha$-loss, a tunable classification loss, to obtain
$(\alpha_D,\alpha_G)$-GANs, parameterized by $(\alpha_D,\alpha_G)\in
(0,\infty]^2$. For sufficiently large number of samples and capacities for G
and D, we show that the resulting non-zero sum game simplifies to minimizing an
$f$-divergence under appropriate conditions on $(\alpha_D,\alpha_G)$. In the
finite sample and capacity setting, we define estimation error to quantify the
gap in the generator's performance relative to the optimal setting with
infinite samples and obtain upper bounds on this error, showing it to be order
optimal under certain conditions. Finally, we highlight the value of tuning
$(\alpha_D,\alpha_G)$ in alleviating training instabilities for the synthetic
2D Gaussian mixture ring and the Stacked MNIST datasets.
( 2
min )
An old problem in multivariate statistics is that linear Gaussian models are
often unidentifiable, i.e. some parameters cannot be uniquely estimated. In
factor (component) analysis, an orthogonal rotation of the factors is
unidentifiable, while in linear regression, the direction of effect cannot be
identified. For such linear models, non-Gaussianity of the (latent) variables
has been shown to provide identifiability. In the case of factor analysis, this
leads to independent component analysis, while in the case of the direction of
effect, non-Gaussian versions of structural equation modelling solve the
problem. More recently, we have shown how even general nonparametric nonlinear
versions of such models can be estimated. Non-Gaussianity is not enough in this
case, but assuming we have time series, or that the distributions are suitably
modulated by some observed auxiliary variables, the models are identifiable.
This paper reviews the identifiability theory for the linear and nonlinear
cases, considering both factor analytic models and structural equation models.
( 2
min )
Contrastively trained encoders have recently been proven to invert the
data-generating process: they encode each input, e.g., an image, into the true
latent vector that generated the image (Zimmermann et al., 2021). However,
real-world observations often have inherent ambiguities. For instance, images
may be blurred or only show a 2D view of a 3D object, so multiple latents could
have generated them. This makes the true posterior for the latent vector
probabilistic with heteroscedastic uncertainty. In this setup, we extend the
common InfoNCE objective and encoders to predict latent distributions instead
of points. We prove that these distributions recover the correct posteriors of
the data-generating process, including its level of aleatoric uncertainty, up
to a rotation of the latent space. In addition to providing calibrated
uncertainty estimates, these posteriors allow the computation of credible
intervals in image retrieval. They comprise images with the same latent as a
given query, subject to its uncertainty. Code is available at
https://github.com/mkirchhof/Probabilistic_Contrastive_Learning
( 2
min )
eXplainable artificial intelligence (XAI) methods have emerged to convert the
black box of machine learning models into a more digestible form. These methods
help to communicate how the model works with the aim of making machine learning
models more transparent and increasing the trust of end-users into their
output. SHapley Additive exPlanations (SHAP) and Local Interpretable Model
Agnostic Explanation (LIME) are two widely used XAI methods particularly with
tabular data. In this commentary piece, we discuss the way the explainability
metrics of these two methods are generated and propose a framework for
interpretation of their outputs, highlighting their weaknesses and strengths.
( 2
min )
We establish matching upper and lower generalization error bounds for
mini-batch Gradient Descent (GD) training with either deterministic or
stochastic, data-independent, but otherwise arbitrary batch selection rules. We
consider smooth Lipschitz-convex/nonconvex/strongly-convex loss functions, and
show that classical upper bounds for Stochastic GD (SGD) also hold verbatim for
such arbitrary nonadaptive batch schedules, including all deterministic ones.
Further, for convex and strongly-convex losses we prove matching lower bounds
directly on the generalization error uniform over the aforementioned class of
batch schedules, showing that all such batch schedules generalize optimally.
Lastly, for smooth (non-Lipschitz) nonconvex losses, we show that full-batch
(deterministic) GD is essentially optimal, among all possible batch schedules
within the considered class, including all stochastic ones.
( 2
min )
We consider a general $p$-norm objective for experimental design problems
that captures some well-studied objectives (D/A/E-design) as special cases. We
prove that a randomized local search approach provides a unified algorithm to
solve this problem for all $p$. This provides the first approximation algorithm
for the general $p$-norm objective, and a nice interpolation of the best known
bounds of the special cases.
( 2
min )
The idea of adversarial learning of regularization functionals has recently
been introduced in the wider context of inverse problems. The intuition behind
this method is the realization that it is not only necessary to learn the basic
features that make up a class of signals one wants to represent, but also, or
even more so, which features to avoid in the representation. In this paper, we
will apply this approach to the problem of source separation by means of
non-negative matrix factorization (NMF) and present a new method for the
adversarial training of NMF bases. We show in numerical experiments, both for
image and audio separation, that this leads to a clear improvement of the
reconstructed signals, in particular in the case where little or no strong
supervision data is available.
( 2
min )
Respiratory syncytial virus (RSV) is one of the most dangerous respiratory
diseases for infants and young children. Due to the nonpharmaceutical
intervention (NPI) imposed in the COVID-19 outbreak, the seasonal transmission
pattern of RSV has been discontinued in 2020 and then shifted months ahead in
2021 in the northern hemisphere. It is critical to understand how COVID-19
impacts RSV and build predictive algorithms to forecast the timing and
intensity of RSV reemergence in post-COVID-19 seasons. In this paper, we
propose a deep coupled tensor factorization machine, dubbed as DeCom, for post
COVID-19 RSV prediction. DeCom leverages tensor factorization and residual
modeling. It enables us to learn the disrupted RSV transmission reliably under
COVID-19 by taking both the regular seasonal RSV transmission pattern and the
NPI into consideration. Experimental results on a real RSV dataset show that
DeCom is more accurate than the state-of-the-art RSV prediction algorithms and
achieves up to 46% lower root mean square error and 49% lower mean absolute
error for country-level prediction compared to the baselines.
( 2
min )
A collaborative research team from the MIT-Takeda Program combined physics and machine learning to characterize rough particle surfaces in pharmaceutical pills and powders.
( 8
min )
Generative AI (GenAI) and large language models (LLMs), such as those available soon via Amazon Bedrock and Amazon Titan are transforming the way developers and enterprises are able to solve traditionally complex challenges related to natural language processing and understanding. Some of the benefits offered by LLMs include the ability to create more capable and […]
( 12
min )
Amazon SageMaker Studio is the first fully integrated development environment (IDE) for ML. It provides a single, web-based visual interface where you can perform all machine learning (ML) development steps required to build, train, tune, debug, deploy, and monitor models. It gives data scientists all the tools you need to take ML models from experimentation […]
( 13
min )
New generations of CPUs offer a significant performance improvement in machine learning (ML) inference due to specialized built-in instructions. Combined with their flexibility, high speed of development, and low operating cost, these general-purpose processors offer an alternative to other existing hardware solutions. AWS, Arm, Meta and others helped optimize the performance of PyTorch 2.0 inference […]
( 6
min )
This post is co-written by Jyoti Sharma and Sharmo Sarkar from Vericast. For any machine learning (ML) problem, the data scientist begins by working with data. This includes gathering, exploring, and understanding the business and technical aspects of the data, along with evaluation of any manipulations that may be needed for the model building process. […]
( 13
min )
Collaboration is key to bringing ideas from lab to life. In the first episode of the #MSRPodcast series “Collaborators,” learn how GitHub’s Kasia Sitkiewicz and Protocol Labs’ Petar Maymounkov are teaming up to make open-source collaborative work better.
The post Collaborators: Gov4git with Petar Maymounkov and Kasia Sitkiewicz appeared first on Microsoft Research.
( 31
min )
This work lists and describes the main recent strategies for building
fixed-length, dense and distributed representations for words, based on the
distributional hypothesis. These representations are now commonly called word
embeddings and, in addition to encoding surprisingly good syntactic and
semantic information, have been proven useful as extra features in many
downstream NLP tasks.
( 2
min )
Adversarial training, which is to enhance robustness against adversarial
attacks, has received much attention because it is easy to generate
human-imperceptible perturbations of data to deceive a given deep neural
network. In this paper, we propose a new adversarial training algorithm that is
theoretically well motivated and empirically superior to other existing
algorithms. A novel feature of the proposed algorithm is to apply more
regularization to data vulnerable to adversarial attacks than other existing
regularization algorithms do. Theoretically, we show that our algorithm can be
understood as an algorithm of minimizing the regularized empirical risk
motivated from a newly derived upper bound of the robust risk. Numerical
experiments illustrate that our proposed algorithm improves the generalization
(accuracy on examples) and robustness (accuracy on adversarial attacks)
simultaneously to achieve the state-of-the-art performance.
( 2
min )
We introduce Robust Exploration via Clustering-based Online Density
Estimation (RECODE), a non-parametric method for novelty-based exploration that
estimates visitation counts for clusters of states based on their similarity in
a chosen embedding space. By adapting classical clustering to the nonstationary
setting of Deep RL, RECODE can efficiently track state visitation counts over
thousands of episodes. We further propose a novel generalization of the inverse
dynamics loss, which leverages masked transformer architectures for multi-step
prediction; which in conjunction with RECODE achieves a new state-of-the-art in
a suite of challenging 3D-exploration tasks in DM-Hard-8. RECODE also sets new
state-of-the-art in hard exploration Atari games, and is the first agent to
reach the end screen in "Pitfall!".
( 2
min )
Gaussian copula mixture models (GCMM) are the generalization of Gaussian
Mixture models using the concept of copula. Its mathematical definition is
given and the properties of likelihood function are studied in this paper.
Based on these properties, extended Expectation Maximum algorithms are
developed for estimating parameters for the mixture of copulas while marginal
distributions corresponding to each component is estimated using separate
nonparametric statistical methods. In the experiment, GCMM can achieve better
goodness-of-fitting given the same number of clusters as GMM; furthermore, GCMM
can utilize unsynchronized data on each dimension to achieve deeper mining of
data.
( 2
min )
AV1, the next-generation video codec, is expanding its reach with today’s release of OBS Studio 29.1. This latest software update adds support for AV1 streaming to YouTube over Enhanced RTMP. All GeForce RTX 40 Series GPUs — including laptop GPUs and the recently launched GeForce RTX 4070 — support real-time AV1 hardware encoding, providing 40% Read article >
( 5
min )
NVIDIA today introduced a wave of cutting-edge AI research that will enable developers and artists to bring their ideas to life — whether still or moving, in 2D or 3D, hyperrealistic or fantastical. Around 20 NVIDIA Research papers advancing generative AI and neural graphics — including collaborations with over a dozen universities in the U.S., Read article >
( 8
min )
Content creator Grant Abbitt embodies selflessness, one of the best qualities that a creative can possess. Passionate about giving back to the creative community, Abbitt offers inspiration, guidance and free education for others in his field through YouTube tutorials.
( 7
min )
One of the most popular models available today is XGBoost. With the ability to solve various problems such as classification and regression, XGBoost has become a popular option that also falls into the category of tree-based models. In this post, we dive deep to see how Amazon SageMaker can serve these models using NVIDIA Triton […]
( 18
min )
Machine learning (ML) helps organizations generate revenue, reduce costs, mitigate risk, drive efficiencies, and improve quality by optimizing core business functions across multiple business units such as marketing, manufacturing, operations, sales, finance, and customer service. With AWS ML, organizations can accelerate the value creation from months to days. Amazon SageMaker Canvas is a visual, point-and-click […]
( 8
min )
Today, we announce the availability of sample notebooks that demonstrate question answering tasks using a Retrieval Augmented Generation (RAG)-based approach with large language models (LLMs) in Amazon SageMaker JumpStart. Text generation using RAG with LLMs enables you to generate domain-specific text outputs by supplying specific external data as part of the context fed to LLMs. […]
( 13
min )
Announcements Big tech must weigh AI’s risks vs. rewards In an interview with the New York Times, Hinton noted the pace of AI advancement is far beyond what he and other tech experts predicted. Hinton said that Google acted very responsibly while he worked on its AI development efforts. His concerns are due to AI’s… Read More »DSC Weekly 2 May 2023 – Big tech must weigh AI’s risks vs. rewards
The post DSC Weekly 2 May 2023 – Big tech must weigh AI’s risks vs. rewards appeared first on Data Science Central.
( 19
min )
Discover the differences between AI, machine learning, and deep learning in this comprehensive guide. Learn how each technology works, their key applications, and the skills required for a career in data science.
The post AI vs Machine Learning vs Deep Learning appeared first on Data Science Central.
( 23
min )
The rapid adoption of smart phones and other mobile platforms has generated an enormous amount of image data. According to Gartner, unstructured data now represents 80–90% of all new enterprise data, but just 18% of organizations are taking advantage of this data. This is mainly due to a lack of expertise and the large amount […]
( 9
min )
Irene Politkoff, Founder and Chief Product Evangelist at semantic modeling tools provider TopQuadrant, posted this description of the large language model (LLM) ChatGPT: “ChatGPT doesn’t access a database of facts to answer your questions. Instead, its responses are based on patterns that it saw in the training data. So ChatGPT is not always trustworthy.” Georgetown… Read More »Can we boost the confidence scores of LLM answers with the help of knowledge graphs?
The post Can we boost the confidence scores of LLM answers with the help of knowledge graphs? appeared first on Data Science Central.
( 20
min )
Customers from Japan to Ecuador and Sweden are using NVIDIA DGX H100 systems like AI factories to manufacture intelligence. They’re creating services that offer AI-driven insights in finance, healthcare, law, IT and telecom — and working to transform their industries in the process. Among the dozens of use cases, one aims to predict how factory Read article >
( 6
min )
Textual backdoor attacks pose a practical threat to existing systems, as they
can compromise the model by inserting imperceptible triggers into inputs and
manipulating labels in the training dataset. With cutting-edge generative
models such as GPT-4 pushing rewriting to extraordinary levels, such attacks
are becoming even harder to detect. We conduct a comprehensive investigation of
the role of black-box generative models as a backdoor attack tool, highlighting
the importance of researching relative defense strategies. In this paper, we
reveal that the proposed generative model-based attack, BGMAttack, could
effectively deceive textual classifiers. Compared with the traditional attack
methods, BGMAttack makes the backdoor trigger less conspicuous by leveraging
state-of-the-art generative models. Our extensive evaluation of attack
effectiveness across five datasets, complemented by three distinct human
cognition assessments, reveals that Figure 4 achieves comparable attack
performance while maintaining superior stealthiness relative to baseline
methods.
( 2
min )
We present a new approach, the Topograph, which reconstructs underlying
physics processes, including the intermediary particles, by leveraging
underlying priors from the nature of particle physics decays and the
flexibility of message passing graph neural networks. The Topograph not only
solves the combinatoric assignment of observed final state objects, associating
them to their original mother particles, but directly predicts the properties
of intermediate particles in hard scatter processes and their subsequent
decays. In comparison to standard combinatoric approaches or modern approaches
using graph neural networks, which scale exponentially or quadratically, the
complexity of Topographs scales linearly with the number of reconstructed
objects.
We apply Topographs to top quark pair production in the all hadronic decay
channel, where we outperform the standard approach and match the performance of
the state-of-the-art machine learning technique.
( 2
min )
With recent advancements in computer vision as well as machine learning (ML),
video-based at-home exercise evaluation systems have become a popular topic of
current research. However, performance depends heavily on the amount of
available training data. Since labeled datasets specific to exercising are
rare, we propose a method that makes use of the abundance of fitness videos
available online. Specifically, we utilize the advantage that videos often not
only show the exercises, but also provide language as an additional source of
information. With push-ups as an example, we show that through the analysis of
subtitle data using natural language processing (NLP), it is possible to create
a labeled (irrelevant, relevant correct, relevant incorrect) dataset containing
relevant information for pose analysis. In particular, we show that irrelevant
clips ($n=332$) have significantly different joint visibility values compared
to relevant clips ($n=298$). Inspecting cluster centroids also show different
poses for the different classes.
( 2
min )
The recent advent of play-to-earn (P2E) systems in massively multiplayer
online role-playing games (MMORPGs) has made in-game goods interchangeable with
real-world values more than ever before. The goods in the P2E MMORPGs can be
directly exchanged with cryptocurrencies such as Bitcoin, Ethereum, or Klaytn
via blockchain networks. Unlike traditional in-game goods, once they had been
written to the blockchains, P2E goods cannot be restored by the game operation
teams even with chargeback fraud such as payment fraud, cancellation, or
refund. To tackle the problem, we propose a novel chargeback fraud prediction
method, PU GNN, which leverages graph attention networks with PU loss to
capture both the players' in-game behavior with P2E token transaction patterns.
With the adoption of modified GraphSMOTE, the proposed model handles the
imbalanced distribution of labels in chargeback fraud datasets. The conducted
experiments on three real-world P2E MMORPG datasets demonstrate that PU GNN
achieves superior performances over previously suggested methods.
( 2
min )
Wind energy is a widely distributed, recyclable and environmentally friendly
energy source that plays an important role in mitigating global warming and
energy shortages. Wind energy's uncertainty and fluctuating nature makes grid
integration of large-scale wind energy systems challenging. Medium-term wind
power forecasts can provide an essential basis for energy dispatch, so accurate
wind power forecasts are essential. Much research has yielded excellent results
in recent years. However, many of them require additional experimentation and
analysis when applied to other data. In this paper, we propose a novel
short-term forecasting framework by tree-structured parzen estimator (TPE) and
decomposition algorithms. This framework defines the TPE-VMD-TFT method for
24-h and 48-h ahead wind power forecasting based on variational mode
decomposition (VMD) and time fusion transformer (TFT). In the Engie wind
dataset from the electricity company in France, the results show that the
proposed method significantly improves the prediction accuracy. In addition,
the proposed framework can be used to other decomposition algorithms and
require little manual work in model training.
( 2
min )
Recent work has shown that simple linear models can outperform several
Transformer based approaches in long term time-series forecasting. Motivated by
this, we propose a Multi-layer Perceptron (MLP) based encoder-decoder model,
Time-series Dense Encoder (TiDE), for long-term time-series forecasting that
enjoys the simplicity and speed of linear models while also being able to
handle covariates and non-linear dependencies. Theoretically, we prove that the
simplest linear analogue of our model can achieve near optimal error rate for
linear dynamical systems (LDS) under some assumptions. Empirically, we show
that our method can match or outperform prior approaches on popular long-term
time-series forecasting benchmarks while being 5-10x faster than the best
Transformer based model.
( 2
min )
Computer vision methods have shown to be effective in classifying garbage
into recycling categories for waste processing, existing methods are costly,
imprecise, and unclear. To tackle this issue, we introduce MWaste, a mobile
application that uses computer vision and deep learning techniques to classify
waste materials as trash, plastic, paper, metal, glass or cardboard. Its
effectiveness was tested on various neural network architectures and real-world
images, achieving an average precision of 92\% on the test set. This app can
help combat climate change by enabling efficient waste processing and reducing
the generation of greenhouse gases caused by incorrect waste disposal.
( 2
min )
Detecting an abrupt distributional shift of the data stream, known as
change-point detection, is a fundamental problem in statistics and signal
processing. We present a new approach for online change-point detection by
training neural networks (NN), and sequentially cumulating the detection
statistics by evaluating the trained discriminating function on test samples by
a CUSUM recursion. The idea is based on the observation that training neural
networks through logistic loss may lead to the log-likelihood function. We
demonstrated the good performance of NN-CUSUM in the detection of
high-dimensional data using both synthetic and real-world data.
( 2
min )
Transfer learning (TL) approaches have shown promising results when handling
tasks with limited training data. However, considerable memory and
computational resources are often required for fine-tuning pre-trained neural
networks with target domain data. In this work, we introduce a novel method for
leveraging pre-trained models for low-resource (music) classification based on
the concept of Neural Model Reprogramming (NMR). NMR aims at re-purposing a
pre-trained model from a source domain to a target domain by modifying the
input of a frozen pre-trained model. In addition to the known,
input-independent, reprogramming method, we propose an advanced reprogramming
paradigm: Input-dependent NMR, to increase adaptability to complex input data
such as musical audio. Experimental results suggest that a neural model
pre-trained on large-scale datasets can successfully perform music genre
classification by using this reprogramming method. The two proposed
Input-dependent NMR TL methods outperform fine-tuning-based TL methods on a
small genre classification dataset.
( 2
min )
Feedback from active galactic nuclei (AGN) and supernovae can affect
measurements of integrated SZ flux of halos ($Y_\mathrm{SZ}$) from CMB surveys,
and cause its relation with the halo mass ($Y_\mathrm{SZ}-M$) to deviate from
the self-similar power-law prediction of the virial theorem. We perform a
comprehensive study of such deviations using CAMELS, a suite of hydrodynamic
simulations with extensive variations in feedback prescriptions. We use a
combination of two machine learning tools (random forest and symbolic
regression) to search for analogues of the $Y-M$ relation which are more robust
to feedback processes for low masses ($M\lesssim 10^{14}\, h^{-1} \, M_\odot$);
we find that simply replacing $Y\rightarrow Y(1+M_*/M_\mathrm{gas})$ in the
relation makes it remarkably self-similar. This could serve as a robust
multiwavelength mass proxy for low-mass clusters and galaxy groups. Our
methodology can also be generally useful to improve the domain of validity of
other astrophysical scaling relations.
We also forecast that measurements of the $Y-M$ relation could provide
percent-level constraints on certain combinations of feedback parameters and/or
rule out a major part of the parameter space of supernova and AGN feedback
models used in current state-of-the-art hydrodynamic simulations. Our results
can be useful for using upcoming SZ surveys (e.g., SO, CMB-S4) and galaxy
surveys (e.g., DESI and Rubin) to constrain the nature of baryonic feedback.
Finally, we find that the an alternative relation, $Y-M_*$, provides
complementary information on feedback than $Y-M$
( 3
min )
The Hopfield model is a paradigmatic model of neural networks that has been
analyzed for many decades in the statistical physics, neuroscience, and machine
learning communities. Inspired by the manifold hypothesis in machine learning,
we propose and investigate a generalization of the standard setting that we
name Random-Features Hopfield Model. Here $P$ binary patterns of length $N$ are
generated by applying to Gaussian vectors sampled in a latent space of
dimension $D$ a random projection followed by a non-linearity. Using the
replica method from statistical physics, we derive the phase diagram of the
model in the limit $P,N,D\to\infty$ with fixed ratios $\alpha=P/N$ and
$\alpha_D=D/N$. Besides the usual retrieval phase, where the patterns can be
dynamically recovered from some initial corruption, we uncover a new phase
where the features characterizing the projection can be recovered instead. We
call this phenomena the learning phase transition, as the features are not
explicitly given to the model but rather are inferred from the patterns in an
unsupervised fashion.
( 2
min )
Recently, quantum classifiers have been found to be vulnerable to adversarial
attacks, in which quantum classifiers are deceived by imperceptible noises,
leading to misclassification. In this paper, we propose the first theoretical
study demonstrating that adding quantum random rotation noise can improve
robustness in quantum classifiers against adversarial attacks. We link the
definition of differential privacy and show that the quantum classifier trained
with the natural presence of additive noise is differentially private. Finally,
we derive a certified robustness bound to enable quantum classifiers to defend
against adversarial examples, supported by experimental results simulated with
noises from IBM's 7-qubits device.
( 2
min )
Semantic segmentation models classifying hyperspectral images (HSI) are
vulnerable to adversarial examples. Traditional approaches to adversarial
robustness focus on training or retraining a single network on attacked data,
however, in the presence of multiple attacks these approaches decrease in
performance compared to networks trained individually on each attack. To combat
this issue we propose an Adversarial Discriminator Ensemble Network (ADE-Net)
which focuses on attack type detection and adversarial robustness under a
unified model to preserve per data-type weight optimally while robustifiying
the overall network. In the proposed method, a discriminator network is used to
separate data by attack type into their specific attack-expert ensemble
network.
( 2
min )
We study reward poisoning attacks on online deep reinforcement learning
(DRL), where the attacker is oblivious to the learning algorithm used by the
agent and the dynamics of the environment. We demonstrate the intrinsic
vulnerability of state-of-the-art DRL algorithms by designing a general,
black-box reward poisoning framework called adversarial MDP attacks. We
instantiate our framework to construct two new attacks which only corrupt the
rewards for a small fraction of the total training timesteps and make the agent
learn a low-performing policy. We provide a theoretical analysis of the
efficiency of our attack and perform an extensive empirical evaluation. Our
results show that our attacks efficiently poison agents learning in several
popular classical control and MuJoCo environments with a variety of
state-of-the-art DRL algorithms, such as DQN, PPO, SAC, etc.
( 2
min )
Voxel-based 3D object classification has been thoroughly studied in recent
years. Most previous methods convert the classic 2D convolution into a 3D form
that will be further applied to objects with binary voxel representation for
classification. However, the binary voxel representation is not very effective
for 3D convolution in many cases. In this paper, we propose a hybrid cascade
architecture for voxel-based 3D object classification. It consists of three
stages composed of fully connected and convolutional layers, dealing with easy,
moderate, and hard 3D models respectively. Both accuracy and speed can be
balanced in our proposed method. By giving each voxel a signed distance value,
an obvious gain regarding the accuracy can be observed. Besides, the mean
inference time can be speeded up hugely compared with the state-of-the-art
point cloud and voxel based methods.
( 2
min )
SplitFed Learning, a combination of Federated and Split Learning (FL and SL),
is one of the most recent developments in the decentralized machine learning
domain. In SplitFed learning, a model is trained by clients and a server
collaboratively. For image segmentation, labels are created at each client
independently and, therefore, are subject to clients' bias, inaccuracies, and
inconsistencies. In this paper, we propose a data quality-based adaptive
averaging strategy for SplitFed learning, called QA-SplitFed, to cope with the
variation of annotated ground truth (GT) quality over multiple clients. The
proposed method is compared against five state-of-the-art model averaging
methods on the task of learning human embryo image segmentation. Our
experiments show that all five baseline methods fail to maintain accuracy as
the number of corrupted clients increases. QA-SplitFed, however, copes
effectively with corruption as long as there is at least one uncorrupted
client.
( 2
min )
A stochastic-gradient-based interior-point algorithm for minimizing a
continuously differentiable objective function (that may be nonconvex) subject
to bound constraints is presented, analyzed, and demonstrated through
experimental results. The algorithm is unique from other interior-point methods
for solving smooth (nonconvex) optimization problems since the search
directions are computed using stochastic gradient estimates. It is also unique
in its use of inner neighborhoods of the feasible region -- defined by a
positive and vanishing neighborhood-parameter sequence -- in which the iterates
are forced to remain. It is shown that with a careful balance between the
barrier, step-size, and neighborhood sequences, the proposed algorithm
satisfies convergence guarantees in both deterministic and stochastic settings.
The results of numerical experiments show that in both settings the algorithm
can outperform a projected-(stochastic)-gradient method.
( 2
min )
Privacy-preserving machine learning solutions have recently gained
significant attention. One promising research trend is using Homomorphic
Encryption (HE), a method for performing computation over encrypted data. One
major challenge in this approach is training HE-friendly, encrypted or
unencrypted, deep CNNs with decent accuracy. We propose a novel training method
for HE-friendly models, and demonstrate it on fundamental and modern CNNs, such
as ResNet and ConvNeXt. After training, we evaluate our models by running
encrypted samples using HELayers SDK and proving that they yield the desired
results. When running on a GPU over the ImageNet dataset, our ResNet-18/50/101
implementations take only 7, 31 and 57 minutes, respectively, which shows that
this solution is practical. Furthermore, we present several insights on
handling the activation functions and skip-connections under HE. Finally, we
demonstrate in an unprecedented way how to perform secure zero-shot prediction
using a CLIP model that we adapted to be HE-friendly.
( 2
min )
The ACM Multimedia 2023 Computational Paralinguistics Challenge addresses two
different problems for the first time in a research competition under
well-defined conditions: In the Emotion Share Sub-Challenge, a regression on
speech has to be made; and in the Requests Sub-Challenges, requests and
complaints need to be detected. We describe the Sub-Challenges, baseline
feature extraction, and classifiers based on the usual ComPaRE features, the
auDeep toolkit, and deep feature extraction from pre-trained CNNs using the
DeepSpectRum toolkit; in addition, wav2vec2 models are used.
( 2
min )
Patient-independent detection of epileptic activities based on visual
spectral representation of continuous EEG (cEEG) has been widely used for
diagnosing epilepsy. However, precise detection remains a considerable
challenge due to subtle variabilities across subjects, channels and time
points. Thus, capturing fine-grained, discriminative features of EEG patterns,
which is associated with high-frequency textural information, is yet to be
resolved. In this work, we propose Scattering Transformer (ScatterFormer), an
invariant scattering transform-based hierarchical Transformer that specifically
pays attention to subtle features. In particular, the disentangled
frequency-aware attention (FAA) enables the Transformer to capture clinically
informative high-frequency components, offering a novel clinical explainability
based on visual encoding of multichannel EEG signals. Evaluations on two
distinct tasks of epileptiform detection demonstrate the effectiveness our
method. Our proposed model achieves median AUCROC and accuracy of 98.14%,
96.39% in patients with Rolandic epilepsy. On a neonatal seizure detection
benchmark, it outperforms the state-of-the-art by 9% in terms of average
AUCROC.
( 2
min )
This paper targets the perceptual task of separating the different
interacting voices, i.e., monophonic melodic streams, in a polyphonic musical
piece. We target symbolic music, where notes are explicitly encoded, and model
this task as a Multi-Trajectory Tracking (MTT) problem from discrete
observations, i.e., notes in a pitch-time space. Our approach builds a graph
from a musical piece, by creating one node for every note, and separates the
melodic trajectories by predicting a link between two notes if they are
consecutive in the same voice/stream. This kind of local, greedy prediction is
made possible by node embeddings created by a heterogeneous graph neural
network that can capture inter- and intra-trajectory information. Furthermore,
we propose a new regularization loss that encourages the output to respect the
MTT premise of at most one incoming and one outgoing link for every node,
favouring monophonic (voice) trajectories; this loss function might also be
useful in other general MTT scenarios. Our approach does not use
domain-specific heuristics, is scalable to longer sequences and a higher number
of voices, and can handle complex cases such as voice inversions and overlaps.
We reach new state-of-the-art results for the voice separation task in
classical music of different styles.
( 2
min )
Graph Neural Networks (GNNs) are a form of deep learning that enable a wide
range of machine learning applications on graph-structured data. The learning
of GNNs, however, is known to pose challenges for memory-constrained devices
such as GPUs. In this paper, we study exact compression as a way to reduce the
memory requirements of learning GNNs on large graphs. In particular, we adopt a
formal approach to compression and propose a methodology that transforms GNN
learning problems into provably equivalent compressed GNN learning problems. In
a preliminary experimental evaluation, we give insights into the compression
ratios that can be obtained on real-world graphs and apply our methodology to
an existing GNN benchmark.
( 2
min )
Kernelized Stein discrepancy (KSD) is a score-based discrepancy widely used
in goodness-of-fit tests. It can be applied even when the target distribution
has an unknown normalising factor, such as in Bayesian analysis. We show
theoretically and empirically that the KSD test can suffer from low power when
the target and the alternative distribution have the same well-separated modes
but differ in mixing proportions. We propose to perturb the observed sample via
Markov transition kernels, with respect to which the target distribution is
invariant. This allows us to then employ the KSD test on the perturbed sample.
We provide numerical evidence that with suitably chosen kernels the proposed
approach can lead to a substantially higher power than the KSD test.
( 2
min )
Experimental data is often comprised of variables measured independently, at
different sampling rates (non-uniform ${\Delta}$t between successive
measurements); and at a specific time point only a subset of all variables may
be sampled. Approaches to identifying dynamical systems from such data
typically use interpolation, imputation or subsampling to reorganize or modify
the training data $\textit{prior}$ to learning. Partial physical knowledge may
also be available $\textit{a priori}$ (accurately or approximately), and
data-driven techniques can complement this knowledge. Here we exploit neural
network architectures based on numerical integration methods and $\textit{a
priori}$ physical knowledge to identify the right-hand side of the underlying
governing differential equations. Iterates of such neural-network models allow
for learning from data sampled at arbitrary time points $\textit{without}$ data
modification. Importantly, we integrate the network with available partial
physical knowledge in "physics informed gray-boxes"; this enables learning
unknown kinetic rates or microbial growth functions while simultaneously
estimating experimental parameters.
( 2
min )
Recent work has shown that simple linear models can outperform several
Transformer based approaches in long term time-series forecasting. Motivated by
this, we propose a Multi-layer Perceptron (MLP) based encoder-decoder model,
Time-series Dense Encoder (TiDE), for long-term time-series forecasting that
enjoys the simplicity and speed of linear models while also being able to
handle covariates and non-linear dependencies. Theoretically, we prove that the
simplest linear analogue of our model can achieve near optimal error rate for
linear dynamical systems (LDS) under some assumptions. Empirically, we show
that our method can match or outperform prior approaches on popular long-term
time-series forecasting benchmarks while being 5-10x faster than the best
Transformer based model.
( 2
min )
Kernelized Stein discrepancy (KSD) is a score-based discrepancy widely used
in goodness-of-fit tests. It can be applied even when the target distribution
has an unknown normalising factor, such as in Bayesian analysis. We show
theoretically and empirically that the KSD test can suffer from low power when
the target and the alternative distribution have the same well-separated modes
but differ in mixing proportions. We propose to perturb the observed sample via
Markov transition kernels, with respect to which the target distribution is
invariant. This allows us to then employ the KSD test on the perturbed sample.
We provide numerical evidence that with suitably chosen kernels the proposed
approach can lead to a substantially higher power than the KSD test.
( 2
min )
We provide an exact expressions for the 1-Wasserstein distance between
independent location-scale distributions. The expressions are represented using
location and scale parameters and special functions such as the standard
Gaussian CDF or the Gamma function. Specifically, we find that the
1-Wasserstein distance between independent univariate location-scale
distributions is equivalent to the mean of a folded distribution within the
same family whose underlying location and scale are equal to the difference of
the locations and scales of the original distributions. A new linear upper
bound on the 1-Wasserstein distance is presented and the asymptotic bounds of
the 1-Wasserstein distance are detailed in the Gaussian case. The effect of
differential privacy using the Laplace and Gaussian mechanisms on the
1-Wasserstein distance is studied using the closed-form expressions and bounds.
( 2
min )
Sparse principal component analysis (SPCA) is widely used for dimensionality
reduction and feature extraction in high-dimensional data analysis. Despite
many methodological and theoretical developments in the past two decades, the
theoretical guarantees of the popular SPCA algorithm proposed by Zou, Hastie &
Tibshirani (2006) are still unknown. This paper aims to address this critical
gap. We first revisit the SPCA algorithm of Zou et al. (2006) and present our
implementation. We also study a computationally more efficient variant of the
SPCA algorithm in Zou et al. (2006) that can be considered as the limiting case
of SPCA. We provide the guarantees of convergence to a stationary point for
both algorithms and prove that, under a sparse spiked covariance model, both
algorithms can recover the principal subspace consistently under mild
regularity conditions. We show that their estimation error bounds match the
best available bounds of existing works or the minimax rates up to some
logarithmic factors. Moreover, we demonstrate the competitive numerical
performance of both algorithms in numerical studies.
( 2
min )
Message Passing Neural Networks (MPNNs) are instances of Graph Neural
Networks that leverage the graph to send messages over the edges. This
inductive bias leads to a phenomenon known as over-squashing, where a node
feature is insensitive to information contained at distant nodes. Despite
recent methods introduced to mitigate this issue, an understanding of the
causes for over-squashing and of possible solutions are lacking. In this
theoretical work, we prove that: (i) Neural network width can mitigate
over-squashing, but at the cost of making the whole network more sensitive;
(ii) Conversely, depth cannot help mitigate over-squashing: increasing the
number of layers leads to over-squashing being dominated by vanishing
gradients; (iii) The graph topology plays the greatest role, since
over-squashing occurs between nodes at high commute (access) time. Our analysis
provides a unified framework to study different recent methods introduced to
cope with over-squashing and serves as a justification for a class of methods
that fall under `graph rewiring'.
( 2
min )
Recovering the latent factors of variation of high dimensional data has so
far focused on simple synthetic settings. Mostly building on unsupervised and
weakly-supervised objectives, prior work missed out on the positive
implications for representation learning on real world data. In this work, we
propose to leverage knowledge extracted from a diversified set of supervised
tasks to learn a common disentangled representation. Assuming each supervised
task only depends on an unknown subset of the factors of variation, we
disentangle the feature space of a supervised multi-task model, with features
activating sparsely across different tasks and information being shared as
appropriate. Importantly, we never directly observe the factors of variations
but establish that access to multiple tasks is sufficient for identifiability
under sufficiency and minimality assumptions. We validate our approach on six
real world distribution shift benchmarks, and different data modalities
(images, text), demonstrating how disentangled representations can be
transferred to real settings.
( 2
min )
We study variance-dependent regret bounds for Markov decision processes
(MDPs). Algorithms with variance-dependent regret guarantees can automatically
exploit environments with low variance (e.g., enjoying constant regret on
deterministic MDPs). The existing algorithms are either variance-independent or
suboptimal. We first propose two new environment norms to characterize the
fine-grained variance properties of the environment. For model-based methods,
we design a variant of the MVP algorithm (Zhang et al., 2021a) and use new
analysis techniques show to this algorithm enjoys variance-dependent bounds
with respect to our proposed norms. In particular, this bound is simultaneously
minimax optimal for both stochastic and deterministic MDPs, the first result of
its kind. We further initiate the study on model-free algorithms with
variance-dependent regret bounds by designing a reference-function-based
algorithm with a novel capped-doubling reference update schedule. Lastly, we
also provide lower bounds to complement our upper bounds.
( 2
min )
A spiking neural network (SNN) equalizer with a decision feedback structure
is applied to an IM/DD link with various parameters. The SNN outperforms linear
and artificial neural network (ANN) based equalizers.
( 2
min )
The goal of this paper is to learn more about how idiomatic information is
structurally encoded in embeddings, using a structural probing method. We
repurpose an existing English verbal multi-word expression (MWE) dataset to
suit the probing framework and perform a comparative probing study of static
(GloVe) and contextual (BERT) embeddings. Our experiments indicate that both
encode some idiomatic information to varying degrees, but yield conflicting
evidence as to whether idiomaticity is encoded in the vector norm, leaving this
an open question. We also identify some limitations of the used dataset and
highlight important directions for future work in improving its suitability for
a probing analysis.
( 2
min )
Experimental data is often comprised of variables measured independently, at
different sampling rates (non-uniform ${\Delta}$t between successive
measurements); and at a specific time point only a subset of all variables may
be sampled. Approaches to identifying dynamical systems from such data
typically use interpolation, imputation or subsampling to reorganize or modify
the training data $\textit{prior}$ to learning. Partial physical knowledge may
also be available $\textit{a priori}$ (accurately or approximately), and
data-driven techniques can complement this knowledge. Here we exploit neural
network architectures based on numerical integration methods and $\textit{a
priori}$ physical knowledge to identify the right-hand side of the underlying
governing differential equations. Iterates of such neural-network models allow
for learning from data sampled at arbitrary time points $\textit{without}$ data
modification. Importantly, we integrate the network with available partial
physical knowledge in "physics informed gray-boxes"; this enables learning
unknown kinetic rates or microbial growth functions while simultaneously
estimating experimental parameters.
( 2
min )
Annealed Importance Sampling (AIS) moves particles along a Markov chain from
a tractable initial distribution to an intractable target distribution. The
recently proposed Differentiable AIS (DAIS) (Geffner and Domke, 2021; Zhang et
al., 2021) enables efficient optimization of the transition kernels of AIS and
of the distributions. However, we observe a low effective sample size in DAIS,
indicating degenerate distributions. We thus propose to extend DAIS by a
resampling step inspired by Sequential Monte Carlo. Surprisingly, we find
empirically-and can explain theoretically-that it is not necessary to
differentiate through the resampling step which avoids gradient variance issues
observed in similar approaches for Particle Filters (Maddison et al., 2017;
Naesseth et al., 2018; Le et al., 2018).
( 2
min )
Organizations are increasingly adopting machine learning (ML) for personnel
assessment. However, concerns exist about fairness in designing and
implementing ML assessments. Supervised ML models are trained to model patterns
in data, meaning ML models tend to yield predictions that reflect subgroup
differences in applicant attributes in the training data, regardless of the
underlying cause of subgroup differences. In this study, we systematically
under- and oversampled minority (Black and Hispanic) applicants to manipulate
adverse impact ratios in training data and investigated how training data
adverse impact ratios affect ML model adverse impact and accuracy. We used
self-reports and interview transcripts from job applicants (N = 2,501) to train
9,702 ML models to predict screening decisions. We found that training data
adverse impact related linearly to ML model adverse impact. However, removing
adverse impact from training data only slightly reduced ML model adverse impact
and tended to negatively affect ML model accuracy. We observed consistent
effects across self-reports and interview transcripts, whether oversampling
real (i.e., bootstrapping) or synthetic observations. As our study relied on
limited predictor sets from one organization, the observed effects on adverse
impact may be attenuated among more accurate ML models.
( 2
min )
This tutorial survey provides an overview of recent non-asymptotic advances
in statistical learning theory as relevant to control and system
identification. While there has been substantial progress across all areas of
control, the theory is most well-developed when it comes to linear system
identification and learning for the linear quadratic regulator, which are the
focus of this manuscript. From a theoretical perspective, much of the labor
underlying these advances has been in adapting tools from modern
high-dimensional statistics and learning theory. While highly relevant to
control theorists interested in integrating tools from machine learning, the
foundational material has not always been easily accessible. To remedy this, we
provide a self-contained presentation of the relevant material, outlining all
the key ideas and the technical machinery that underpin recent results. We also
present a number of open problems and future directions.
( 2
min )
Conditional Average Treatment Effects (CATE) estimation is one of the main
challenges in causal inference with observational data. In addition to Machine
Learning based-models, nonparametric estimators called meta-learners have been
developed to estimate the CATE with the main advantage of not restraining the
estimation to a specific supervised learning method. This task becomes,
however, more complicated when the treatment is not binary as some limitations
of the naive extensions emerge. This paper looks into meta-learners for
estimating the heterogeneous effects of multi-valued treatments. We consider
different meta-learners, and we carry out a theoretical analysis of their error
upper bounds as functions of important parameters such as the number of
treatment levels, showing that the naive extensions do not always provide
satisfactory results. We introduce and discuss meta-learners that perform well
as the number of treatments increases. We empirically confirm the strengths and
weaknesses of those methods with synthetic and semi-synthetic datasets.
( 2
min )
A crucial challenge in reinforcement learning is to reduce the number of
interactions with the environment that an agent requires to master a given
task. Transfer learning proposes to address this issue by re-using knowledge
from previously learned tasks. However, determining which source task qualifies
as the most appropriate for knowledge extraction, as well as the choice
regarding which algorithm components to transfer, represent severe obstacles to
its application in reinforcement learning. The goal of this paper is to address
these issues with modular multi-source transfer learning techniques. The
proposed techniques automatically learn how to extract useful information from
source tasks, regardless of the difference in state-action space and reward
function. We support our claims with extensive and challenging cross-domain
experiments for visual control.
( 2
min )
We propose the first online quantum algorithm for zero-sum games with $\tilde
O(1)$ regret under the game setting. Moreover, our quantum algorithm computes
an $\varepsilon$-approximate Nash equilibrium of an $m \times n$ matrix
zero-sum game in quantum time $\tilde O(\sqrt{m+n}/\varepsilon^{2.5})$,
yielding a quadratic improvement over classical algorithms in terms of $m, n$.
Our algorithm uses standard quantum inputs and generates classical outputs with
succinct descriptions, facilitating end-to-end applications. As an application,
we obtain a fast quantum linear programming solver. Technically, our online
quantum algorithm "quantizes" classical algorithms based on the optimistic
multiplicative weight update method. At the heart of our algorithm is a fast
quantum multi-sampling procedure for the Gibbs sampling problem, which may be
of independent interest.
( 2
min )
We analyze the generalization ability of joint-training meta learning
algorithms via the Gibbs algorithm. Our exact characterization of the expected
meta generalization error for the meta Gibbs algorithm is based on symmetrized
KL information, which measures the dependence between all meta-training
datasets and the output parameters, including task-specific and meta
parameters. Additionally, we derive an exact characterization of the meta
generalization error for the super-task Gibbs algorithm, in terms of
conditional symmetrized KL information within the super-sample and super-task
framework introduced in Steinke and Zakynthinou (2020) and Hellstrom and Durisi
(2022) respectively. Our results also enable us to provide novel
distribution-free generalization error upper bounds for these Gibbs algorithms
applicable to meta learning.
( 2
min )
Many techniques in machine learning attempt explicitly or implicitly to infer
a low-dimensional manifold structure of an underlying physical phenomenon from
measurements without an explicit model of the phenomenon or the measurement
apparatus. This paper presents a cautionary tale regarding the discrepancy
between the geometry of measurements and the geometry of the underlying
phenomenon in a benign setting. The deformation in the metric illustrated in
this paper is mathematically straightforward and unavoidable in the general
case, and it is only one of several similar effects. While this is not always
problematic, we provide an example of an arguably standard and harmless data
processing procedure where this effect leads to an incorrect answer to a
seemingly simple question. Although we focus on manifold learning, these issues
apply broadly to dimensionality reduction and unsupervised learning.
( 2
min )
The learnable, linear neural network layers between tensor power spaces of
$\mathbb{R}^{n}$ that are equivariant to the orthogonal group, $O(n)$, the
special orthogonal group, $SO(n)$, and the symplectic group, $Sp(n)$, were
characterised in arXiv:2212.08630. We present an algorithm for multiplying a
vector by any weight matrix for each of these groups, using category theoretic
constructions to implement the procedure. We achieve a significant reduction in
computational cost compared with a naive implementation by making use of
Kronecker product matrices to perform the multiplication. We show that our
approach extends to the symmetric group, $S_n$, recovering the algorithm of
arXiv:2303.06208 in the process.
( 2
min )
Neural network model compression techniques can address the computation issue
of deep neural networks on embedded devices in industrial systems. The
guaranteed output error computation problem for neural network compression with
quantization is addressed in this paper. A merged neural network is built from
a feedforward neural network and its quantized version to produce the exact
output difference between two neural networks. Then, optimization-based methods
and reachability analysis methods are applied to the merged neural network to
compute the guaranteed quantization error. Finally, a numerical example is
proposed to validate the applicability and effectiveness of the proposed
approach.
( 2
min )
We introduce a new computational framework for estimating parameters in
generalized generalized linear models (GGLM), a class of models that extends
the popular generalized linear models (GLM) to account for dependencies among
observations in spatio-temporal data. The proposed approach uses a monotone
operator-based variational inequality method to overcome non-convexity in
parameter estimation and provide guarantees for parameter recovery. The results
can be applied to GLM and GGLM, focusing on spatio-temporal models. We also
present online instance-based bounds using martingale concentrations
inequalities. Finally, we demonstrate the performance of the algorithm using
numerical simulations and a real data example for wildfire incidents.
( 2
min )
Gradient-boosted decision trees (GBDT) are widely used and highly effective
machine learning approach for tabular data modeling. However, their complex
structure may lead to low robustness against small covariate perturbation in
unseen data. In this study, we apply one-hot encoding to convert a GBDT model
into a linear framework, through encoding of each tree leaf to one dummy
variable. This allows for the use of linear regression techniques, plus a novel
risk decomposition for assessing the robustness of a GBDT model against
covariate perturbations. We propose to enhance the robustness of GBDT models by
refitting their linear regression forms with $L_1$ or $L_2$ regularization.
Theoretical results are obtained about the effect of regularization on the
model performance and robustness. It is demonstrated through numerical
experiments that the proposed regularization approach can enhance the
robustness of the one-hot-encoded GBDT models.
( 2
min )
In this paper, a computationally efficient data-driven hybrid automaton model
is proposed to capture unknown complex dynamical system behaviors using
multiple neural networks. The sampled data of the system is divided by valid
partitions into groups corresponding to their topologies and based on which,
transition guards are defined. Then, a collection of small-scale neural
networks that are computationally efficient are trained as the local dynamical
description for their corresponding topologies. After modeling the system with
a neural-network-based hybrid automaton, the set-valued reachability analysis
with low computation cost is provided based on interval analysis and a split
and combined process. At last, a numerical example of the limit cycle is
presented to illustrate that the developed models can significantly reduce the
computational cost in reachable set computation without sacrificing any
modeling precision.
( 2
min )
Federated learning (FL) is an emerging technique that trains massive and
geographically distributed edge data while maintaining privacy. However, FL has
inherent challenges in terms of fairness and computational efficiency due to
the rising heterogeneity of edges, and thus usually results in sub-optimal
performance in recent state-of-the-art (SOTA) solutions. In this paper, we
propose a Customized Federated Learning (CFL) system to eliminate FL
heterogeneity from multiple dimensions. Specifically, CFL tailors personalized
models from the specially designed global model for each client jointly guided
by an online trained model-search helper and a novel aggregation algorithm.
Extensive experiments demonstrate that CFL has full-stack advantages for both
FL training and edge reasoning and significantly improves the SOTA performance
w.r.t. model accuracy (up to 7.2% in the non-heterogeneous environment and up
to 21.8% in the heterogeneous environment), efficiency, and FL fairness.
( 2
min )
Semantic knowledge of part-part and part-whole relationships in assemblies is
useful for a variety of tasks from searching design repositories to the
construction of engineering knowledge bases. In this work we propose that the
natural language names designers use in Computer Aided Design (CAD) software
are a valuable source of such knowledge, and that Large Language Models (LLMs)
contain useful domain-specific information for working with this data as well
as other CAD and engineering-related tasks.
In particular we extract and clean a large corpus of natural language part,
feature and document names and use this to quantitatively demonstrate that a
pre-trained language model can outperform numerous benchmarks on three
self-supervised tasks, without ever having seen this data before. Moreover, we
show that fine-tuning on the text data corpus further boosts the performance on
all tasks, thus demonstrating the value of the text data which until now has
been largely ignored. We also identify key limitations to using LLMs with text
data alone, and our findings provide a strong motivation for further work into
multi-modal text-geometry models.
To aid and encourage further work in this area we make all our data and code
publicly available.
( 2
min )
In this paper we present the first version of ganX -- generate artificially
new XRF, a Python library to generate X-ray fluorescence Macro maps (MA-XRF)
from a coloured RGB image. To do that, a Monte Carlo method is used, where each
MA-XRF pixel signal is sampled out of an XRF signal probability function. Such
probability function is computed using a database of couples (pigment
characteristic XRF signal, RGB), by a weighted sum of such pigment XRF signal
by proximity of the image RGB to the pigment characteristic RGB. The library is
released to PyPi and the code is available open source on GitHub.
( 2
min )
Distributed learning paradigms, such as federated or decentralized learning,
allow a collection of agents to solve global learning and optimization problems
through limited local interactions. Most such strategies rely on a mixture of
local adaptation and aggregation steps, either among peers or at a central
fusion center. Classically, aggregation in distributed learning is based on
averaging, which is statistically efficient, but susceptible to attacks by even
a small number of malicious agents. This observation has motivated a number of
recent works, which develop robust aggregation schemes by employing robust
variations of the mean. We present a new attack based on sensitivity curve
maximization (SCM), and demonstrate that it is able to disrupt existing robust
aggregation schemes by injecting small, but effective perturbations.
( 2
min )
This paper provides answers to an open problem: given a nonlinear data-driven
dynamical system model, e.g., kernel conditional mean embedding (CME) and
Koopman operator, how can one propagate the ambiguity sets forward for multiple
steps? This problem is the key to solving distributionally robust control and
learning-based control of such learned system models under a data-distribution
shift. Different from previous works that use either static ambiguity sets,
e.g., fixed Wasserstein balls, or dynamic ambiguity sets under known piece-wise
linear (or affine) dynamics, we propose an algorithm that exactly propagates
ambiguity sets through nonlinear data-driven models using the Koopman operator
and CME, via the kernel maximum mean discrepancy geometry. Through both
theoretical and numerical analysis, we show that our kernel ambiguity sets are
the natural geometric structure for the learned data-driven dynamical system
models.
( 2
min )
The success of the Adam optimizer on a wide array of architectures has made
it the default in settings where stochastic gradient descent (SGD) performs
poorly. However, our theoretical understanding of this discrepancy is lagging,
preventing the development of significant improvements on either algorithm.
Recent work advances the hypothesis that Adam and other heuristics like
gradient clipping outperform SGD on language tasks because the distribution of
the error induced by sampling has heavy tails. This suggests that Adam
outperform SGD because it uses a more robust gradient estimate. We evaluate
this hypothesis by varying the batch size, up to the entire dataset, to control
for stochasticity. We present evidence that stochasticity and heavy-tailed
noise are not major factors in the performance gap between SGD and Adam.
Rather, Adam performs better as the batch size increases, while SGD is less
effective at taking advantage of the reduction in noise. This raises the
question as to why Adam outperforms SGD in the full-batch setting. Through
further investigation of simpler variants of SGD, we find that the behavior of
Adam with large batches is similar to sign descent with momentum.
( 2
min )
This paper introduces JaxPruner, an open-source JAX-based pruning and sparse
training library for machine learning research. JaxPruner aims to accelerate
research on sparse neural networks by providing concise implementations of
popular pruning and sparse training algorithms with minimal memory and latency
overhead. Algorithms implemented in JaxPruner use a common API and work
seamlessly with the popular optimization library Optax, which, in turn, enables
easy integration with existing JAX based libraries. We demonstrate this ease of
integration by providing examples in four different codebases: Scenic, t5x,
Dopamine and FedJAX and provide baseline experiments on popular benchmarks.
( 2
min )
In recent years, there has been a surge in effort to formalize notions of
fairness in machine learning. We focus on clustering -- one of the fundamental
tasks in unsupervised machine learning. We propose a new axiom that captures
proportional representation fairness (PRF). We make a case that the concept
achieves the raison d'{\^{e}}tre of several existing concepts in the literature
in an arguably more convincing manner. Our fairness concept is not satisfied by
existing fair clustering algorithms. We design efficient algorithms to achieve
PRF both for unconstrained and discrete clustering problems.
( 2
min )
In documents and graphics, contours are a popular format to describe specific
shapes. For example, in the True Type Font (TTF) file format, contours describe
vector outlines of typeface shapes. Each contour is often defined as a sequence
of points. In this paper, we tackle the contour completion task. In this task,
the input is a contour sequence with missing points, and the output is a
generated completed contour. This task is more difficult than image completion
because, for images, the missing pixels are indicated. Since there is no such
indication in the contour completion task, we must solve the problem of missing
part detection and completion simultaneously. We propose a Transformer-based
method to solve this problem and show the results of the typeface contour
completion.
( 2
min )
This study aims to alleviate the trade-off between utility and privacy in the
task of differentially private clustering. Existing works focus on simple
clustering methods, which show poor clustering performance for non-convex
clusters. By utilizing Morse theory, we hierarchically connect the Gaussian
sub-clusters to fit complex cluster distributions. Because differentially
private sub-clusters are obtained through the existing methods, the proposed
method causes little or no additional privacy loss. We provide a theoretical
background that implies that the proposed method is inductive and can achieve
any desired number of clusters. Experiments on various datasets show that our
framework achieves better clustering performance at the same privacy level,
compared to the existing methods.
( 2
min )
In 1-bit matrix completion, the aim is to estimate an underlying low-rank
matrix from a partial set of binary observations. We propose a novel method for
1-bit matrix completion called MMGN. Our method is based on the
majorization-minimization (MM) principle, which yields a sequence of standard
low-rank matrix completion problems in our setting. We solve each of these
sub-problems by a factorization approach that explicitly enforces the assumed
low-rank structure and then apply a Gauss-Newton method. Our numerical studies
and application to a real-data example illustrate that MMGN outputs comparable
if not more accurate estimates, is often significantly faster, and is less
sensitive to the spikiness of the underlying matrix than existing methods.
( 2
min )
In recent years, product categorisation has been a common issue for
E-commerce companies who have utilised machine learning to categorise their
products automatically. In this study, we propose an ensemble approach, using a
combination of different models to separately predict each product's category,
subcategory, and colour before ultimately combining the resultant predictions
for each product. With the aforementioned approach, we show that an average
F1-score of 0.82 can be achieved using a combination of XGBoost and k-nearest
neighbours to predict said features.
( 2
min )
Innovative Electronic Design Automation (EDA) solutions are important to meet
the design requirements for increasingly complex electronic devices. Verilog, a
hardware description language, is widely used for the design and verification
of digital circuits and is synthesized using specific EDA tools. However,
writing code is a repetitive and time-intensive task. This paper proposes,
primarily, a novel deep learning framework for training a Verilog
autocompletion model and, secondarily, a Verilog dataset of files and snippets
obtained from open-source repositories. The framework involves integrating
models pretrained on general programming language data and finetuning them on a
dataset curated to be similar to a target downstream task. This is validated by
comparing different pretrained models trained on different subsets of the
proposed Verilog dataset using multiple evaluation metrics. These experiments
demonstrate that the proposed framework achieves better BLEU, ROUGE-L, and chrF
scores by 9.5%, 6.7%, and 6.9%, respectively, compared to a model trained from
scratch.
( 2
min )
While the use of the Internet of Things is becoming more and more popular,
many security vulnerabilities are emerging with the large number of devices
being introduced to the market. In this environment, IoT device identification
methods provide a preventive security measure as an important factor in
identifying these devices and detecting the vulnerabilities they suffer from.
In this study, we present a method that identifies devices in the Aalto dataset
using Long short-term memory (LSTM)
( 2
min )
In online forums like Reddit, users share their experiences with medical
conditions and treatments, including making claims, asking questions, and
discussing the effects of treatments on their health. Building systems to
understand this information can effectively monitor the spread of
misinformation and verify user claims. The Task-8 of the 2023 International
Workshop on Semantic Evaluation focused on medical applications, specifically
extracting patient experience- and medical condition-related entities from user
posts on social media. The Reddit Health Online Talk (RedHot) corpus contains
posts from medical condition-related subreddits with annotations characterizing
the patient experience and medical conditions. In Subtask-1, patient experience
is characterized by personal experience, questions, and claims. In Subtask-2,
medical conditions are characterized by population, intervention, and outcome.
For the automatic extraction of patient experiences and medical condition
information, as a part of the challenge, we proposed language-model-based
extraction systems that ranked $3^{rd}$ on both subtasks' leaderboards. In this
work, we describe our approach and, in addition, explore the automatic
extraction of this information using domain-specific language models and the
inclusion of external knowledge.
( 2
min )
This paper assesses the reliability of the RemOve-And-Retrain (ROAR)
protocol, which is used to measure the performance of feature importance
estimates. Our findings from the theoretical background and empirical
experiments indicate that attributions that possess less information about the
decision function can perform better in ROAR benchmarks, conflicting with the
original purpose of ROAR. This phenomenon is also observed in the recently
proposed variant RemOve-And-Debias (ROAD), and we propose a consistent trend of
blurriness bias in ROAR attribution metrics. Our results caution against
uncritical reliance on ROAR metrics.
( 2
min )
Current dialogue research primarily studies pairwise (two-party)
conversations, and does not address the everyday setting where more than two
speakers converse together. In this work, we both collect and evaluate
multi-party conversations to study this more general case. We use the LIGHT
environment to construct grounded conversations, where each participant has an
assigned character to role-play. We thus evaluate the ability of language
models to act as one or more characters in such conversations. Models require
two skills that pairwise-trained models appear to lack: (1) being able to
decide when to talk; (2) producing coherent utterances grounded on multiple
characters. We compare models trained on our new dataset to existing
pairwise-trained dialogue models, as well as large language models with
few-shot prompting. We find that our new dataset, MultiLIGHT, which we will
publicly release, can help bring significant improvements in the group setting.
( 2
min )
To improve the recognition ability of computer-aided breast mass
classification among mammographic images, in this work we explore the
state-of-the-art classification networks to develop an ensemble mechanism.
First, the regions of interest (ROIs) are obtained from the original dataset,
and then three models, i.e., XceptionNet, DenseNet, and EfficientNet, are
trained individually. After training, we ensemble the mechanism by summing the
probabilities outputted from each network which enhances the performance up to
5%. The scheme has been validated on a public dataset and we achieved accuracy,
precision, and recall 88%, 85%, and 76% respectively.
( 2
min )
Gradient-boosted decision trees (GBDT) are widely used and highly effective
machine learning approach for tabular data modeling. However, their complex
structure may lead to low robustness against small covariate perturbation in
unseen data. In this study, we apply one-hot encoding to convert a GBDT model
into a linear framework, through encoding of each tree leaf to one dummy
variable. This allows for the use of linear regression techniques, plus a novel
risk decomposition for assessing the robustness of a GBDT model against
covariate perturbations. We propose to enhance the robustness of GBDT models by
refitting their linear regression forms with $L_1$ or $L_2$ regularization.
Theoretical results are obtained about the effect of regularization on the
model performance and robustness. It is demonstrated through numerical
experiments that the proposed regularization approach can enhance the
robustness of the one-hot-encoded GBDT models.
( 2
min )
Annealed Importance Sampling (AIS) moves particles along a Markov chain from
a tractable initial distribution to an intractable target distribution. The
recently proposed Differentiable AIS (DAIS) (Geffner and Domke, 2021; Zhang et
al., 2021) enables efficient optimization of the transition kernels of AIS and
of the distributions. However, we observe a low effective sample size in DAIS,
indicating degenerate distributions. We thus propose to extend DAIS by a
resampling step inspired by Sequential Monte Carlo. Surprisingly, we find
empirically-and can explain theoretically-that it is not necessary to
differentiate through the resampling step which avoids gradient variance issues
observed in similar approaches for Particle Filters (Maddison et al., 2017;
Naesseth et al., 2018; Le et al., 2018).
( 2
min )
We detail an approach to develop Stein's method for bounding integral metrics
on probability measures defined on a Riemannian manifold $\mathbf M$. Our
approach exploits the relationship between the generator of a diffusion on
$\mathbf M$ with target invariant measure and its characterising Stein
operator. We consider a pair of such diffusions with different starting points,
and through analysis of the distance process between the pair, derive Stein
factors, which bound the solution to the Stein equation and its derivatives.
The Stein factors contain curvature-dependent terms and reduce to those
currently available for $\mathbb R^m$, and moreover imply that the bounds for
$\mathbb R^m$ remain valid when $\mathbf M$ is a flat manifold
( 2
min )
Conditional Average Treatment Effects (CATE) estimation is one of the main
challenges in causal inference with observational data. In addition to Machine
Learning based-models, nonparametric estimators called meta-learners have been
developed to estimate the CATE with the main advantage of not restraining the
estimation to a specific supervised learning method. This task becomes,
however, more complicated when the treatment is not binary as some limitations
of the naive extensions emerge. This paper looks into meta-learners for
estimating the heterogeneous effects of multi-valued treatments. We consider
different meta-learners, and we carry out a theoretical analysis of their error
upper bounds as functions of important parameters such as the number of
treatment levels, showing that the naive extensions do not always provide
satisfactory results. We introduce and discuss meta-learners that perform well
as the number of treatments increases. We empirically confirm the strengths and
weaknesses of those methods with synthetic and semi-synthetic datasets.
( 2
min )
We introduce a new computational framework for estimating parameters in
generalized generalized linear models (GGLM), a class of models that extends
the popular generalized linear models (GLM) to account for dependencies among
observations in spatio-temporal data. The proposed approach uses a monotone
operator-based variational inequality method to overcome non-convexity in
parameter estimation and provide guarantees for parameter recovery. The results
can be applied to GLM and GGLM, focusing on spatio-temporal models. We also
present online instance-based bounds using martingale concentrations
inequalities. Finally, we demonstrate the performance of the algorithm using
numerical simulations and a real data example for wildfire incidents.
( 2
min )
Variational Bayes is a popular method for approximate inference but its
derivation can be cumbersome. To simplify the process, we give a 3-step recipe
to identify the posterior form by explicitly looking for linearity with respect
to expectations of well-known distributions. We can then directly write the
update by simply ``reading-off'' the terms in front of those expectations. The
recipe makes the derivation easier, faster, shorter, and more general.
( 2
min )
In 1-bit matrix completion, the aim is to estimate an underlying low-rank
matrix from a partial set of binary observations. We propose a novel method for
1-bit matrix completion called MMGN. Our method is based on the
majorization-minimization (MM) principle, which yields a sequence of standard
low-rank matrix completion problems in our setting. We solve each of these
sub-problems by a factorization approach that explicitly enforces the assumed
low-rank structure and then apply a Gauss-Newton method. Our numerical studies
and application to a real-data example illustrate that MMGN outputs comparable
if not more accurate estimates, is often significantly faster, and is less
sensitive to the spikiness of the underlying matrix than existing methods.
( 2
min )
This tutorial survey provides an overview of recent non-asymptotic advances
in statistical learning theory as relevant to control and system
identification. While there has been substantial progress across all areas of
control, the theory is most well-developed when it comes to linear system
identification and learning for the linear quadratic regulator, which are the
focus of this manuscript. From a theoretical perspective, much of the labor
underlying these advances has been in adapting tools from modern
high-dimensional statistics and learning theory. While highly relevant to
control theorists interested in integrating tools from machine learning, the
foundational material has not always been easily accessible. To remedy this, we
provide a self-contained presentation of the relevant material, outlining all
the key ideas and the technical machinery that underpin recent results. We also
present a number of open problems and future directions.
( 2
min )
Message Passing Neural Networks (MPNNs) are instances of Graph Neural
Networks that leverage the graph to send messages over the edges. This
inductive bias leads to a phenomenon known as over-squashing, where a node
feature is insensitive to information contained at distant nodes. Despite
recent methods introduced to mitigate this issue, an understanding of the
causes for over-squashing and of possible solutions are lacking. In this
theoretical work, we prove that: (i) Neural network width can mitigate
over-squashing, but at the cost of making the whole network more sensitive;
(ii) Conversely, depth cannot help mitigate over-squashing: increasing the
number of layers leads to over-squashing being dominated by vanishing
gradients; (iii) The graph topology plays the greatest role, since
over-squashing occurs between nodes at high commute (access) time. Our analysis
provides a unified framework to study different recent methods introduced to
cope with over-squashing and serves as a justification for a class of methods
that fall under `graph rewiring'.
( 2
min )
The learnable, linear neural network layers between tensor power spaces of
$\mathbb{R}^{n}$ that are equivariant to the orthogonal group, $O(n)$, the
special orthogonal group, $SO(n)$, and the symplectic group, $Sp(n)$, were
characterised in arXiv:2212.08630. We present an algorithm for multiplying a
vector by any weight matrix for each of these groups, using category theoretic
constructions to implement the procedure. We achieve a significant reduction in
computational cost compared with a naive implementation by making use of
Kronecker product matrices to perform the multiplication. We show that our
approach extends to the symmetric group, $S_n$, recovering the algorithm of
arXiv:2303.06208 in the process.
( 2
min )
We study variance-dependent regret bounds for Markov decision processes
(MDPs). Algorithms with variance-dependent regret guarantees can automatically
exploit environments with low variance (e.g., enjoying constant regret on
deterministic MDPs). The existing algorithms are either variance-independent or
suboptimal. We first propose two new environment norms to characterize the
fine-grained variance properties of the environment. For model-based methods,
we design a variant of the MVP algorithm (Zhang et al., 2021a) and use new
analysis techniques show to this algorithm enjoys variance-dependent bounds
with respect to our proposed norms. In particular, this bound is simultaneously
minimax optimal for both stochastic and deterministic MDPs, the first result of
its kind. We further initiate the study on model-free algorithms with
variance-dependent regret bounds by designing a reference-function-based
algorithm with a novel capped-doubling reference update schedule. Lastly, we
also provide lower bounds to complement our upper bounds.
( 2
min )
Many techniques in machine learning attempt explicitly or implicitly to infer
a low-dimensional manifold structure of an underlying physical phenomenon from
measurements without an explicit model of the phenomenon or the measurement
apparatus. This paper presents a cautionary tale regarding the discrepancy
between the geometry of measurements and the geometry of the underlying
phenomenon in a benign setting. The deformation in the metric illustrated in
this paper is mathematically straightforward and unavoidable in the general
case, and it is only one of several similar effects. While this is not always
problematic, we provide an example of an arguably standard and harmless data
processing procedure where this effect leads to an incorrect answer to a
seemingly simple question. Although we focus on manifold learning, these issues
apply broadly to dimensionality reduction and unsupervised learning.
( 2
min )
Wartella and AI reinvigorate a White Stripes classic, exploring AI’s role in music video creation.
( 6
min )
Source Manufacturers often turn to digitalization strategies to improve their competitiveness, address labor shortages, and boost productivity. These efforts are driven by a desire to stay ahead of the game rather than simply defend against the competition. However, moving to the front foot regarding generated data unlocks waves of innovation — creating fast, bold, competitive,… Read More »3 Major Benefits Data Collection Brings To The Manufacturing Process
The post 3 Major Benefits Data Collection Brings To The Manufacturing Process appeared first on Data Science Central.
( 21
min )
Sponsored Post Attend the Data Science Symposium 2022 on November 8 The Center for Business Analytics at the University of Cincinnati will present its annual Data Science Symposium 2022 on November 8. This all day in-person event will have three featured speakers and two tech talk tracks with four concurrent presentations in each track. The […]
The post Attend the Data Science Symposium 2022, November 8 in Cincinnati appeared first on Machine Learning Mastery.
( 10
min )
